.arunim.fyiThis post was co-written with Brandon, all errors mine until he tells me that he wants credit for the errors.
On the difference between AGI, ASI, and other powerful AIs, and why conflating them all under the umbrella of AGI has been quite bad
Back when the labs were further away from AGI, they were less precise: they’d say “AGI has the potential to cause extinction” when what they meant was something more like “AGI has the potential to lead to recursive self-improvement and this can then lead to ASI, which has the potential to cause extinction”. Sensibly so, as who knew how long it’d take to achieve AGI, and if there was indeed a hard takeoff, the interim period wouldn’t matter so much.
Fortunately, it seems like we aren’t in as fast of a takeoff, but are still speeding along towards AGI and recursive self-improvement.
Unfortunately, this has caused confusion in two of the most important places on Planet Earth: Washington, D.C. and twitter dot com. People often claim “powerful just isn’t dangerous, because {human analogy}” while what they really believe is “I don’t think we’ll go from AGI to ASI, and AGI isn’t dangerous because {human analogy}”
People also often have very different things in mind when they say AGI: Definitions have been made time and time again but it hasn’t led to meaningful convergence.
Even when there’s some decree or disclaimer in the introduction of a paper or post that says “we’re defining AGI as having passed capabilities x, y, z in one system” this is rarely fully internalized; they still have this deep-rooted AGI definition. I think this especially applies to words like TAI, which I think are often implicitly replaced by internal definitions of AGI/ASI.
There are many capabilities thresholds that matter potentially more so than these acronyms: can a “PhD student with a wet lab” build a bioweapon, percentages of remote workers able to be automated, non-ML SWE work, ML capabilities sufficient for recursive self improvement, etc.
Bad things will happen if people continue to be imprecise:
a16z will point to a very capable system and say “AGI is here and it hasn’t killed us all, see we’re fine, the safetyists were wrong all along, dynamism goes brrr”. Then Congress deregulates AI and there’s [insert bad event].
Actual AGI will be here before ASI, and OpenAI will say “AGI is here and it hasn’t killed us all, see we’re fine, we solved safety”. Then they develop ASI and it’s “like lights out for all of us” (- Sam Altman).
When someone says “AGI is 5 years away,” we won’t know what they mean. Creates less ideal community discourse. Everyone talks about METR’s forecasting work but then immediately says “this means AGI is x years away” instead of saying “this means x% remote jobs will be automated in y years”. Off the top of my head I couldn’t tell you what METR’s results are except maybe some vague “AGI 2029”.
Social/psychological effects - public fatigue, boy who cried wolf (already present in some capacity). Especially true if timelines are less short than we think.
Legal/discourse confusion - People on the hill use the same terms to mean different things, people don’t accurately understand legislation or people don’t pass legislation. Similar to SB 1047: “Politicians were not understanding the bill as being about extinction threats; they were understanding the bill as being about regulatory capture of a normal budding technology industry.” - Nate Soares. By calling them all “AGI,” we muddy the waters for policymakers and make it harder to craft appropriate responses to each category of risk.
A not horrible warning shot happens and the public believes that “this is the worst that can happen, since this is already AGI”. See (directionally, though maybe not exactly) COVID.
Good scenarios will happen if people are precise:
a16z will only be able to say “AI has now automated 80% of remote work, however there are obvious other stages because AI systems still can’t build ferraris, not capable of recursive self improvement”
See above for OpenAI
People will say “I think we’re 5 years away from 80% of remote work being automated” which is easily interpretable
Public will more clearly hear how these thresholds have shifted have improved and this will be more salient because they will realize how far away they are from e.g. losing their job. Much easier to tell my Mom “experts forecast we’re 5 years away from 80% remote work automation” than to tell my Mom “AGI is 5 years away”. Of course you could still do both in today’s world, it’s just harder than it needs to be, and then my Mom will start to hear words like AGI in a year and won’t know what it means.
People on the Hill will refer to the same capabilities marks and not talk past each other.
Warning shot happens and we can rigorously point to other marks that have not been hit, like RSI, which AGI might not necessarily have.
This is Actually Pretty Easy to Fix
The solution here is straightforward: stop using “AGI” in professional contexts. Just don’t do it. Use specific capability thresholds instead.
Organizations like METR, Anthropic, and OpenAI should standardize around concrete benchmarks. Instead of “AGI by 2027,” publish forecasts like “75% remote work automation by 2027” or “bioweapon-capable AI by 2029.”
Researchers get clearer, more actionable predictions. Policymakers get concrete targets to regulate around. The public gets information they can actually use to prepare for economic disruption.
Even the labs benefit because they can’t get away with moving goalposts as easily. Much harder to say “this isn’t the dangerous AI we were worried about” when the threshold was “systems capable of recursive self-improvement” rather than “AGI.”
Afterthought
Not having good definitions for AGI creates even more of a race to the bottom for timelines, where people are incentivized to have lower timelines because cool factor, urgency etc. If you instead referenced marks, and people have to boldly claim “80% remote work by x date” then you’re more incentivized to get your timeline perfect. Also AGI timelining gives severe rise to the conjunction fallacy which you eliminate when you rid the term.