Anthropic’s AI systems could lead to catastrophe or entirely wipe out humanity, it has warned as it seeks billions of dollars to keep building them.
uncertain
it → lead → them
The company gave the warning in newly released documents ahead of its initial public offering, which will see it listed on the stock market in what could be a record breaking deal.
uncertain
what → give → market
The documents pitch investing in the company as a way to bet on AI’s transformation of the world – but warns, at the same time, that it could destroy it.
uncertain
it → pitch → it
In a long list of potential dangers for the company, it warns that its systems could “pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponised, or allowed to operate with a high degree of autonomy”.
uncertain
systems → warn → autonomy
The IPO documents point to a whole range of ways that Anthropic’s systems could pose a threat.
uncertain
systems → point → threat
They include models that have already been shown to be able to help people “develop chemical, biological, or other dangerous weapons”, and the use of its models in “high-risk or sensitive contexts, such as healthcare, scientific and biomedical research, drug discovery and development, legal, finance, government, or other critical decision-making environments”.
asserted
people → include → healthcare
“Our AI models may themselves cause harm by producing undesirable outputs or behaving in unintended, misaligned, or autonomous ways, including by taking unintended or harmful actions that no user or third party directed,” Anthropic warns.
uncertain
Anthropic → cause → that
“These risks may increase as our models become more capable and are granted greater autonomy, permissions, and integration with external systems.
uncertain
models → increase → systems
While Anthropic says that it is working both to understand and limit those risks, it warns that they are unpredictable by nature.
asserted
they → say → nature
“Although we continue to study why these behaviours emerge, our understanding continues to evolve, and we may be unable to anticipate or understand such behaviours in the future,” it warns, noting that the problematic behaviours “may manifest in unpredictable ways”.
uncertain
behaviours → continue → ways
Anthropic has already seen its systems behaving differently when they know they are being tested, for instance, and so even detailed oversight may be “unreliable”.
uncertain
oversight → see → instance
And they could also develop entirely new kinds of capabilities that Anthropic may not be aware of, leaving them especially unable to deal with them.
uncertain
Anthropic → develop → them
The documents also point to less dramatic but potentially more immediate risks.
asserted
documents → point → risks
It notes that “the long-term effects of regular AI interaction on mental health, cognitive development, and social behaviour are not yet fully understood”, for instance.
asserted
effects → note → instance
In all, Anthropic wrote 94 pages of warnings, almost a quarter of the entire document.
asserted
Anthropic → write → document
As with all IPO documents, many of those warnings are specifically focused on the ways that it would affect investors, and the health of the company.
asserted
it → focus → company
In its warning that its systems could cause harm in unintended ways on its own, for instance, Anthropic says such an incident “could adversely affect our reputation and brand, business, financial condition, and results of operations”.
uncertain
incident → cause → operations
The same document does however make grand claims about the ways that AI could help humanity.
uncertain
AI → make → humanity
It could for instance accelerate scientific discovery, “potentially curing diseases that have plagued humanity for millennia, including cancer, Alzheimer’s disease, heart disease, rare genetic diseases and the major infectious diseases”, Anthropic claims.
uncertain
Anthropic → accelerate → cancer
The flotation will allow Anthropic’s investors, which include Amazon and Google, to sell some of their shares.
asserted
which → allow → shares
But it will also raise new money through the deal, which it said would be used to fund a range of investments.
asserted
it → raise → investments
Some of that will be used for “investments in compute and infrastructure”, it said.
asserted
it → use → compute
In practice, that is likely to mean the building and purchasing of new data centres and the other expensive hardware that is required to run AI systems.
asserted
that → mean → systems
The Aporia
