Photo via Noahpinion

Should we "pace" AI self-improvement?

Read the original at Noahpinion ↗
Cite

Fist, T. (2026, September 4). Should we "pace" AI self-improvement? Noahpinion. https://www.noahpinion.blog/p/should-we-pace-ai-self-improvement

Dated when The Aporia collected the article, usually within hours of publication. Check the original for its exact date and byline.

Noahpinion · collected 2026-09-04 · by Tim Fist commentary

Quick Summary

The author of this opinion piece argues that advanced artificial intelligence (AI) could potentially design superviruses that threaten human existence, and that intentionally slowing down the development of AI capabilities may be necessary to prevent this outcome. The author mentions a recent open letter signed by over 1,300 employees across US frontier AI companies calling for government support in developing tools to "pace" the development of automated AI. The author notes that leading AI labs, including OpenAI and Anthropic, have expressed concerns about the risks of fully automating AI research and development (R&D). The piece will explore the idea of "pacing" and whether it is a viable solution to address potential risks posed by advanced AI.
Written by AI on 2026-09-05, from this article's own text only (the story summary below draws on all the coverage).
See how they differ ↓

AI analysis

Story summary

On September 8, Jacob Coxon, a 27-year-old researcher who previously worked at both OpenAI and Anthropic, resigned from Anthropic, warning that the companies are racing towards developing self-improving superintelligence and potentially putting human lives at risk. Coxon's resignation drew significant attention online and in media outlets like CNN and BBC News, with experts such as Geoffrey Hinton estimating a 10% chance of AI causing human extinction within this decade. Notably, several current Anthropic employees echoed Coxon’s concerns on social media platforms, emphasizing the urgency of addressing AI safety issues before it is too late. Despite these warnings, some in Silicon Valley and industry leaders remain skeptical about the immediate threat posed by AI advancements.

Written for “AI Safety And Human Extinction Warnings” on 2026-11-20, grounded in this article and the 29 other(s) covering the same event.

How they differ

21 outlets covered “AI Safety And Human Extinction Warnings”.

What each outlet says

One AI summary per outlet of its latest article on this story, written from that article alone and checked against it. Open the article for the outlet's own words.

NoahpinionThis outlet4 Sep

The author of this opinion piece argues that advanced artificial intelligence (AI) could potentially design superviruses that threaten human existence, and that intentionally slowing down the development of AI capabilities may be necessary to prevent this outcome. The author mentions a recent open letter signed by over 1,300 employees across US frontier AI companies calling for government support in developing tools…

Should we "pace" AI self-improvement? (this article)
Semafor12 Sep

Reed discusses the overreaction to Jacob Coxon's resignation from Anthropic, where he warned about potential human extinction due to AI. The article clarifies that fear around AI should not be conflated with concerns about environmental costs, hacking, and job market disruption. Instead of panicking, lawmakers and tech leaders need to address these issues precisely without hindering beneficial AI innovation or…

We have nothing to fear but AI fear itself →4 articles from Semafor on this story
BBC15 Sep

At a recent Goldman Sachs conference in San Francisco, tech executives addressed concerns raised by Jacob Coxon, an Anthropic researcher who recently resigned due to safety fears about advanced AI. Coxon warned that these technologies could potentially destroy humanity and "gamble with our lives." This came amidst other high-profile resignations from both Anthropic and OpenAI over similar safety concerns. While some…

Dramatic insider warnings over AI fall flat with some in Silicon Valley →4 articles from BBC on this story
The Straits Times9 Sep

Jacob Coxon, a 27-year-old artificial intelligence researcher, has left Anthropic after three years at the company and accused both OpenAI and Anthropic of recklessly racing to develop self-improving superintelligence, which could pose an existential risk to humanity. According to Coxon, neither company is acting responsibly in their pursuit of AI development. Anthropic's safety executive, Evan Hubinger, agrees that…

‘They’re playing with our lives’: AI researcher quits Anthropic →2 articles from The Straits Times on this story
The Free Press10 Sep

A former employee of AI companies OpenAI and Anthropic warns that their pursuit of superintelligence could be catastrophic, echoing a prediction made by one of the researchers himself. According to Jacob Coxon's resignation post on X, these companies are racing towards developing systems that can hack anything and acquire real power, which he believes poses an existential threat. Coxon claims that people within…

Yes, AI Might Really Kill Us All →2 articles from The Free Press on this story
CBS News11 Sep

Jacob Coxon, a former researcher at Anthropic, resigned from his position on Tuesday, accusing the company and rival OpenAI of recklessly developing advanced AI models that pose a threat to humanity. There is no specific data or numbers mentioned in the article about the risks posed by AI. Coxon's resignation comes after he expressed concerns that companies are prioritizing competition over safety, leading him to…

Extended Interview: Ex-Anthropic researcher Jacob Coxon, who warns AI could destroy humanity →2 articles from CBS News on this story
Show 14 more outlets
South China Morning Post7 Sep

UN High Commissioner for Human Rights Volker Turk warned that artificial intelligence poses an "existential risk" to humanity in a speech to the 47-member UN council in Geneva on Monday. He called for "cast-iron guarantees" to ensure the safety and security of AI, citing concerns from industry insiders about its potential dangers. According to Turk's office, potential risks include disruptions to critical…

UN rights chief warns AI could pose ‘existential’ risk to humanity’ →
Platformer9 Sep

AI lab leaders are urging urgent action on regulating artificial intelligence, while U.S. political figures remain uninterested. Previously dismissed as a fringe concern, existential risks posed by AI have now become a major topic of discussion within the industry. Despite recent antitrust court losses for tech companies, there has been little concrete change, and calls for slowing down AI development continue to…

The AI warnings are coming from inside the lab →
Daily Mail9 Sep

Jacob Coxon, a researcher at Anthropic, has quit his job over concerns that big tech firms are racing to develop AI systems that could potentially wipe out humanity by the end of the decade. He claims that neither OpenAI nor Anthropic is acting responsibly in regards to AI safety. Coxon warns that superintelligent AI could become "superhuman systems" with the power to hack anything and acquire resources, posing a…

AI could 'kill all humans' by the end of the decade, Anthropic researcher warns - as he quits over 'out of control' superintelligence race →
New York Post9 Sep

Jacob Coxon, a researcher with 3 years of experience in pre-training research at Anthropic and OpenAI, has abruptly quit his job, warning that AI "could kill us all by the end of the decade". He claims both companies are racing towards creating self-improving superintelligence without acting responsibly. According to Coxon, executives and senior researchers are privately expressing fear about the technology's…

Anthropic researcher abruptly quits, warning AI ‘could kill us all’ →
ABC News (AU)9 Sep

Jacob Coxon, an Anthropic researcher, is leaving his position at the company after stating that many AI developers, including those at Anthropic, believe that artificial intelligence has the potential to be catastrophic by the end of the decade. This concern is shared by hundreds of employees from leading tech firms who recently urged the US government to support international efforts to manage advanced AI…

Anthropic researcher says some developers believe AI 'could kill us all' →
Dawn9 Sep

Jacob Coxon, a researcher at Anthropic, has resigned citing concerns over the irresponsible development of artificial intelligence (AI) by companies. He claims that labs are "gambling with our lives" in their rush to develop self-improving superintelligence. Coxon, who previously worked at OpenAI and has expertise in training AI models, warns that such technology poses a significant threat to humanity, potentially…

Anthropic researcher resigns, says AI labs are ‘gambling with our lives’ →
Time10 Sep

Jacob Coxon, a 27-year-old AI researcher who spent three years working on powerful AI systems at OpenAI and Anthropic, has resigned from his position citing concerns about the industry's lack of responsibility and control over its rapid development. He claims that neither company is acting responsibly in their pursuit of self-improving superintelligence. The key number mentioned in the article is 90 million views…

He Helped Build Powerful AI at OpenAI and Anthropic. Now He's Afraid It Could Kill Us →
Mother Jones11 Sep

Former Anthropic researcher Jacob Coxon has resigned, warning that companies are "gambling with our lives" by developing AI models that could reach "superintelligence". Coxon's colleague Evan Hubinger also expressed concern on social media, stating that staffers at Anthropic genuinely believe AI could pose a threat to human existence. Meanwhile, leading companies continue to rapidly improve their AI models, with…

Anthropic Staffers Again Sound the Alarm on AI Catastrophe →
Zeteo11 Sep

On Tuesday, 27-year-old AI researcher Jacob Coxon started a Twitter thread warning of the existential threat posed by AI systems, claiming they could "kill us all" by the decade's end. A group of 9 top AI researchers and executives, including Coxon, have come out of the woodwork to sound the alarm bells about the dangers of AI over the last 72 hours, with some expressing surprise that their warnings were not taken…

The AI Apocalypse? Here Are 9 AI Experts Warning It Could Kill Us All Very Soon →
ABC News (US)11 Sep

A researcher at Anthropic, a company working with artificial intelligence, is leaving the field and issuing a warning that AI may become uncontrollable by late 2027. No specific details are provided on why the researcher is leaving or what actions they believe need to be taken to mitigate this risk. OpenAI's chief scientist also warns of rising machine intelligence, saying no one is prepared for it.

WATCH: Insiders' new AI warning →
NPR12 Sep

Jacob Coxon, a British AI researcher who recently resigned from Anthropic, has warned about the risks associated with rapidly advancing AI technology, suggesting that companies like Anthropic and OpenAI are prioritizing speed over safety. Coxon's concerns stem from witnessing how quickly AI systems are improving while the ability to control them remains unclear. His warnings have sparked discussions among fellow…

Why are the people building the most powerful AI so worried about what it could do? →
CBC13 Sep

An outgoing researcher at Anthropic warned last week that AI could pose an existential threat, potentially "killing all humans" by the end of this decade. This warning highlights concerns among experts about the rapid advancement and potential loss of control over increasingly sophisticated AI systems. The International AI Safety Report describes a "loss of control" scenario where AI operates beyond human oversight…

Could AI really 'kill all humans'? Breaking down the dire warnings about the technology's existential risk →
The Dispatch15 Sep

In this podcast episode, Steve Hayes discusses with guests Jonah Goldberg, Mike Warren, and Klon Kitchen about concerns raised by Jacob Coxen, an Anthropic researcher who quit over fears AI might pose existential risks within a decade. The discussion centers on the ethical implications and potential dangers of rapid advancements in artificial intelligence technology.

Will Artificial Intelligence ‘Kill Us All?’ →
The Guardian15 Sep

Stuart Russell discusses the recent controversies surrounding artificial intelligence, including Jacob Coxon's resignation from Anthropic and concerns raised by the OpenAI/Hugging Face incident. The Anthropic CEO, Dario Amodei, has proposed a strategy called "pacing the frontier" to address these issues without halting progress entirely. This involves third-party evaluators within companies, common safety standards…

AI safety requires more than just slowing our pace | Stuart Russell →

Where they agree and split

Agreed

AI · cited, from two or more outlets

  • AI researchers are worried that the development of self-improving superintelligence could pose a significant threat to humanity if not managed responsibly.14
Quoted from the coverage (1)

On Tuesday, AI researcher Jacob Coxon announced his resignation from Anthropic in a mega-viral post on X: “I spent the last three years doing pretraining research at both OpenAI and Anthropic,” he began. (Pretraining is an important part of the training process for AI models like Claude and GPT-6.)

5 outlets

Framing

How the first headlines put it

  • The Straits Times focuses on the researcher's resignation and concerns about AI safety, while Daily Mail emphasizes the potential for AI to 'kill all humans' by the end of the decade.12
  • CBS News presents a balanced view of the researcher's warnings, whereas The Free Press uses more dramatic language like 'Yes, AI Might Really Kill Us All'.56
The full story: what happened, how it changed, and all 30 articles →

Compared with similar articles

This article reads leans left and hedges 37% of its claims. Each row says how that neighbour differs.
New York Post
Hedges 2 of 12 claims Outlet record 66/100
ABC News (AU)
Hedges 2 of 4 claims Outlet record 67/100
“Article A is a personal opinion piece, while Article B reports on an Anthropic researcher's claim about AI developers' beliefs and recent industry developments.”
Platformer
Leans left Hedges 0 of 4 claims Outlet record 96/100
“The articles are discussing different topics related to AI, but do not describe the same specific happening or incident”

Story

AI Safety And Human Extinction Warnings
Technology · 30 article(s) covering the same event. Compare every outlet's account →
How this article was measured signals, leaning charts, publisher record, byline, topics and claims

Signals How these are calculated →

Claims extracted
101
sentences that state something as happening or true
Uncertain
37%
37 of 101 hedged ("reportedly", "may", "according to")
Leaning
Leans left
expected in commentary, which argues a position · beta estimate
Publisher signals
not measured
these signals count newsroom corrections; Commentary has none to count
Outlets on this story
30
Technology
Same framing elsewhere
1
articles that frame the event the way this one does
Analyzed 2026-09-05 · how these are computed

How this is being covered How these are calculated →

Article leaning vs. publisher reliability
Source leaning vs. consistency

Publisher

Noahpinion · 59 article(s) · 0 correction(s) detected
No corrections detected for this publisher. That may mean careful reporting, or simply that nothing has been checked.

Who wrote this

Tim Fist
2 article(s) here · 1 carrying a prediction
On the other hand, I do think there’s a substantial chance that AI will kill most people on Earth within the next decade or two, by designing superviruses.
2026-09-04 · mixed framing · Should we "pace" AI self-improvement?
Rapid progress towards fully automated AI R&D has empirical support, but it’s less clear how much it will accelerate AI capabilities or pose severe risks.
2026-08-25 · mixed framing · 23 low-regret recommendations for AI policy
Also by Tim Fist
Nothing else under this byline is closely related to this article, so these are simply their most recent.

Topics

Anthropic OpenAI the Institute for Progress

Subjects

Anthropic ORG · 1× OpenAI ORG · 1× Saif Khan PERSON · 1× Sam Altman PERSON · 1× Tim Fist PERSON · 1× the Institute for Progress ORG · 1×

Narrative

Incentivizing the reallocation of resources away from those severely risky activities and towards two ends: Accelerating the diffusion of AI capabilities (including via developing applied AI tools like AlphaFold). Accelerating research that would make further AI R&D automation safer, either by: Improving the safety of AI models directly (e.g., via developing AI control protocols), or Boosting societal resilience to make the negative consequences of new AI capabilities less acute (e.g., via using AI to patch open-source code vulnerabilities).
framing: mixed · carried by 1 article(s) · first seen 2026-09-05
On the other hand, I do think there’s a substantial chance that AI will kill most people on Earth within the next decade or two, by designing superviruses.
2026-09-05 · Noahpinion
Should we "pace" AI self-improvement? · mixed framing

Claims (101 extracted, 37 hedged)

On one hand, I love AI technology. asserted
I → love → technology
On the other hand, I do think there’s a substantial chance that AI will kill most people on Earth within the next decade or two, by designing superviruses. asserted
AI → think → superviruses
AI is already capable of designing viruses not found in nature, so this isn’t a sci-fi scenario. asserted
this → design → nature
Whether these superviruses would be designed and unleashed by nihilistic human individuals, doomsday cults, or rogue AI agents themselves might end up being a secondary question. uncertain
designed → design → individuals
We know we have nihilistic human individuals who might decide to destroy civilization in a fit of depression or pique. uncertain
who → know → depression
We know we have doomsday cults. asserted
we → know → cults
The will to destroy humanity exists, and sufficiently capable AI will probably provide a way, if sufficient precautions are not taken. asserted
precautions → destroy → way
But right now, nobody really knows what precautions will be sufficient. asserted
precautions → know → ?
One idea — promoted by the big AI labs themselves! — is to intentionally slow down the development of AI capabilities. asserted
idea → promote → capabilities
This could conceivably buy us time to take other precautions, such as improved security around bio-labs, better AI alignment, and so on. uncertain
This → buy → labs
Intentionally slowing AI development is called “pacing”. asserted
slowing → slow → development
The biggest question facing the “pacing” debate right now is whether to curb the use of AI to design better AI — often called “recusive self-improvement”, or “RSI” for short. asserted
question → face → AI
I haven’t waded into the pacing debate myself, but as a start, I thought it would be interesting to publish the thoughts of the good folks at the Institute for Progress, whose judgement I generally trust. asserted
I → wad → judgement
Part 1 (today’s post) covers how seriously we should take this possibility of RSI, and whether it justifies slowing down frontier AI development. asserted
it → cover → development
Part 2 will cover policy recommendations. asserted
Part → cover → recommendations
If you work in US policy and would like to connect with the authors, you can reach Tim Fist at tim.fist@ifp.org and Saif Khan at saif@ifp.org. asserted
you → work → saif@ifp.org
Frontier AI companies are racing to automate the development of AI, but they seem increasingly worried about what will happen if they succeed. asserted
they → race → AI
More than 1,300 employees across every US frontier AI company recently signed an open letter calling for the government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” asserted
government → sign → development
The official OpenAI and Anthropic accounts tweeted messages in support of the letter, and the same day Sam Altman told an interviewer “we may have to pace the rate of AI development.” uncertain
we → tweet → development
But before considering whether the letter is relevant for government policy, we have to answer two questions: What does “pacing” actually mean? asserted
pacing → consider → What
And does the argument for it stand up to scrutiny? asserted
argument → stand → scrutiny
In our view, the letter is implicitly arguing three things: Frontier AI companies are close to fully automating AI R&D. Automating AI R&D would pose serious risks. asserted
Automating → argue → risks
Building the option to “pace” — i.e., somehow slow down progress toward fully automated AI R&D — is a good way to address those risks. asserted
Building → build → risks
Slowing down AI progress should not be taken lightly: Advances in AI could unlock massive societal benefits, from new cures for diseases to abundant robotic labor. uncertain
Advances → slow → labor
Yet if the AI researchers and CEOs are right about automating AI R&D — both that they could do it and that it would be extremely risky — the right tradeoffs for policymakers might look very different when we get there. uncertain
we → automate → policymakers
With the right preparation, we might be able to manage the risks of automated AI R&D while having AI’s capabilities progress faster and diffuse more broadly than they do today. uncertain
they → manage → R&D
So, despite substantial uncertainty, we believe the US should take low-regret policy actions now to prepare for a possible future in which serious risks from automated AI R&D require some form of “pacing.” asserted
risks → believe → pacing
Here we’ll explain why, including what makes us take the open letter’s claims seriously and our principles for choosing policies with minimal downside if the risks prove overblown. uncertain
risks → explain → downside
In the next post, we’ll provide a detailed list of specific policy recommendations. asserted
we → provide → recommendations
Are frontier AI companies close to fully automating AI R&D? asserted
companies → automate → R&D
Frontier AI companies are racing to automate AI R&D. Sam Altman, for example, stated last year that OpenAI aimed to have a “true automated researcher” by March 2028, and Anthropic’s leaders have made similar predictions.1 asserted
leaders → race → March
But how do those goals stack up to reality? asserted
goals → stack → reality
One way of answering is to look at how models are getting better over time at AI R&D. asserted
models → answer → R&D.
You can break down the skills required for an AI model to do AI R&D into two broad categories: software engineering, where the model writes code for research experiments and training runs, and research taste, where AI models decide which experiments are worth trying. asserted
experiments → break → experiments
For software engineering, AI capabilities appear to be increasing exponentially. asserted
capabilities → appear → engineering
When AI models are evaluated against how long it would take humans to complete the same engineering tasks — so-called “time horizons” — their capabilities seem to be doubling every 7 months.2 These capability improvements apply to the software engineering tasks required for AI R&D. asserted
improvements → evaluate → R&D.
In a long-running experiment, researchers at Anthropic have found that their models now significantly outperform humans under a fixed time budget on an AI R&D task focused on speeding up AI model training. asserted
models → run → training
For research taste, frontier AI models also show signs of fast improvement, though the evidence is less clear. asserted
evidence → show → improvement
According to Anthropic’s research, its models have rapidly become proficient at solving “open-ended” tasks that Anthropic’s technical staff work on, where the model must solve problems with no clear specification by generating multiple approaches and then deciding among them. uncertain
model → accord → them
In a separate open-ended research project where the AI was tasked with proposing and testing hypotheses about an open problem in AI safety, Anthropic’s models significantly outperformed two human researchers (97% performance improvement vs. 23%) when given a similar time budget (5 to 7 days). asserted
models → task → budget
…and 61 more, not listed.
Give feedback · no sign-in needed

No sign-in needed for now. Your note is sent without a name, so it cannot be answered.

History Support