To be in San Francisco, one of America’s greatest cities, right now is to be steamrolled by the artificial intelligence wave that is breaking over the world.
asserted
that → steamroll → world
When you fly in to Melbourne or Sydney, you’re greeted by billboards selling fancy furniture, sex shops and blockbuster TV shows.
asserted
you → fly → furniture
Cruise around the San Francisco Bay Area for a couple of days and the advertisements are selling the benefits of AI – from “right-sizing” your business (sacking humans and hiring AI) to curing major diseases, smarter investing through to soft hacking government.
asserted
advertisements → cruise → government
This great Pacific city is in the grip of an AI fever-dream and it’s like Jack Kerouac embracing Arthur Rimbaud.
asserted
Kerouac → ’ → Rimbaud
Deranged, but oh-so-real.
There is still a chance to manage AI, but the window is closing.
asserted
window → be → AI
I’m on a study tour this week paid for by the Australian Council on AI Strategy (ACAIS is a newish not-for-profit and non-partisan body funded by philanthropists) and I’ve joined the heads of major Australian foreign affairs and economic think tanks, government officials and a handful of scientists in visiting frontier AI giant bigwigs including OpenAI, maker of ChatGPT, and Anthropic, the maker of Claude (journalists were excluded from the latter meeting).
asserted
journalists → ’m → meeting
We also heard from and met with AI safety experts, and heard conflicting versions of what the AI revolution is going to do to Australia and the world.
asserted
revolution → hear → Australia
The utopian version is that AI will usher in a global golden age.
asserted
AI → usher → age
The dystopian version is that once frontier AI models (the ones that are most advanced, like those made by Anthropic and OpenAI) achieve “full” recursive self-improvement [RSI] in the next year or so, AI will build better, more sophisticated versions of itself without any human assistance or oversight at all (AI already does most of this work itself, but not all of it).
asserted
AI → make → it
The pessimists suggest there is a 10-50 per cent chance that humanity will be wiped out as a species in the next couple of years.
asserted
humanity → suggest → years
This is just another data point to consider.
asserted
This → consider → ?
As one of the AI safety experts put it: “if you have things you want to do in your life, do them by 2027”.
asserted
you → put → 2027
Or as another participant put it during our Chatham House discussions, there are “multiple Manhattan Projects being run by private companies” right now.
asserted
participant → put → companies
The US is leading the world on AI.
asserted
US → lead → AI
It’s several months ahead of China.
asserted
It → ’ → China
It’s also the wild west of regulation.
asserted
It → ’ → regulation
Thankfully there are people sounding the alarm, including Karl Berzins, an Australian engineer who co-founded the SF-based FAR.AI, a not-for-profit focused on safety and heading off the worst of what could be coming down the pipe.
uncertain
what → be → pipe
As Berzins puts it, Australia’s federal government understands the risks and the belated announcement of an AI safety institute confirms this.
asserted
announcement → put → this
“My worry is whether we can move fast enough.
asserted
we → move → ?
Australia traditionally likes to play it safe and that just won’t work here.
asserted
that → like → it
I would love us to lead ... if we can’t lead, however, we should at least try to keep up!”
Bringing AI training onshore from a company such as Anthropic would give us leverage, but it’s not guaranteed.
asserted
it → love → leverage
And the Albanese government isn’t spending nearly enough money.
asserted
government → spend → money
“We [FAR.AI] recently signed a compute deal worth around $50 million over three years.
asserted
We → sign → years
Talk to the pessimists, and all bets are off as to whether humanity itself will survive once RSI is achieved – because once it’s achieved, the AI agents we currently ask to plan our trips overseas, choose a recipe or help with work will move from coding and maths-smarts to (a now out-of-date term) artificial general intelligence and then on to superintelligence.
asserted
we → talk → superintelligence
At that point, in the same way we hardly ever notice when stepping on or around an ant, humanity’s survival depends primarily on whether it suits the most advanced AI models to look down for a moment while passing us by.
asserted
it → notice → us
At this point, you probably think I’m insane.
asserted
I → think → point
I checked in with several sources in the Australian government’s intelligence community during the course of the week and was reassured (I think) that they believe humanity has until about 2030, rather than 2027, to sort out this existential threat and head it off.
asserted
humanity → check → it
But they agreed the threat was real and imminent.
asserted
threat → agree → ?
In an Australian context, the AI debate is domestic, focused on the fight over copyright payments for artists whose work is used to train AI (which happens now in the US under fair-use laws which differ from Australia), the introduction of criminal penalties and mandatory reporting requirements for cyber incidents such after ChatGPT’s Medicare hack, the belated recent creation of an Australian AI Safety Institute and whether or not Anthropic might invest in AI training technology on local soil.
uncertain
Anthropic → focus → soil
This is, to put it mildly, small beer.
asserted
This → put → it
Australia is focused on the wrong questions.
asserted
Australia → focus → questions
To put it another way: it’s not really going to matter whether Cate Blanchett or Jimmy Barnes receives a copyright payment for helping to train AI models if humanity ends in the next year.
asserted
humanity → put → year
Getting some of Anthropic’s computer power to Australia, which will give us some degree of leverage over policymaking and development, matters far more.
asserted
which → get → policymaking
That’s not guaranteed.
asserted
That → guarantee → ?
In 2021, Anthropic’s AI tech could not multiply a two-digit problem.
uncertain
tech → multiply → problem
In 2026, it solved the Navier Stokes mathematical equation.
asserted
it → solve → equation
If AI superintelligence occurs (likely, but not definite), faster-than-light travel could be possible in the next decade (extremely unlikely, but not impossible).
uncertain
travel → occur → decade
Some AI organisations are so worried about the power of the tech, they’re running tabletop exercises that war game what would happen if, for example, a powerful AI model accidentally revealed the location of every submarine off the west coast of the US, both enemies and allies, one day soon.
asserted
model → run → US
AI agents – that is, about 1200 copies of the company’s AI models – could cure all major diseases that afflict humanity in the next few years.
uncertain
that → cure → years
Or they could end humanity.
uncertain
they → end → humanity
…and 12 more, not listed.
The Aporia
