.arunim.fyiStatus: very-rough draft
This tweet from Aidan McLaughlin, circa September 2024, feels pretty representative of the vibe at the time.
asi feels closer than agi if that makes sense
And the hype only intensified after we saw o3-preview.
My rough model at the time was “oh okay they’re just going to RLVR all the way up to ASI somehow”, but it seems like that hasn’t turned out to be the case. More and more compute has been applied to the RL part of training, and while the returns have been large, they don’t seem to be large enough to directly get us to completely automating AI R&D. Nor have they generalized in the strong ways that might have been expected.
I think I’m getting a bit off track with this line of thought, though. The actual thing that I wanted to discuss or write about here is different models for takeoff or recursive self-improvement, and to actually examine how many of them, or which of them, are realistic given the evidence that we have accumulated so far. What I meant to convey with the RLVR point was that there was some feeling at the time in September 2024, that the path to ASI was indeed clear and it would simply be to scale up RL training. I think that is just not the case now for a variety of reasons. There are still a variety of approaches that could be plausible for recursive self-improvement, and I’d like to examine what they are because, in my view, the risks that we have to consider from AI development are quite path dependent on how exactly are we getting to ASI.
What is RSI?? I think it means humans exit the loop of AI development. But like … how much? it could be the case that humans are cheaper than automated alternatives for a large variety of physical labor for some time. And still, we might attribute >90% of advances in AI R&D to AI systems, and 100% of the technical or conceptual advances. Is that RSI?
Specifically, the concrete form of this claim is something like: You could automate the entire AI company. The automated AI company could have a business plan, could sell products, could “earn its way”, and spend some of the profits that it makes on physical labor that is done by humans. Is that RSI?
For the stronger definition of RSI, which requires that humans are completely out of the loop of AI development, this is not RSI. However, this company would still move much, much faster. Specifically, it would move closer to the upper bound of the rate of progress than human-run AI companies would.
Pure-software
We hold inputs roughly fixed, or capped under some humanly-achievable threshold, and AI of various forms is able to improve the efficiency of all the different parts of the chain that it controls.
I think this gets at a clear or an interesting dynamic where the parts of AI R&D that get automated immediately compress to small parts. As Tyler Cowen says, all the other parts become the bottlenecks. The human-run parts become the bottlenecks. The rate of progress kind of increases as a result of the automation and then quickly hits some barrier because of the next thing that is not yet automated.
It seems pretty likely to me that we do not live in a world where the genuinely pure software thing is actually feasible. That is, I don’t think we live in a world where AI systems can make advances in AI that are large enough that the next set of AI systems which have made use of these advances are able to find the next set of advances, and so on and so forth.
Part of the reason might be that good ideas are getting harder to find, and the inputs that are necessary, or relatively the inputs that are necessary for each discovery of an equivalent magnitude, are kind of exponentially larger. The gains you get from that discovery are not exponentially larger.
Most of the gains, or the way that the gains have been able to contribute to the next generation, is not so much directly, but rather indirectly via the economy, via diffusion, via revenues of the companies growing massively. Notably, however, the revenues have been increasing exponentially. If the revenues were increasing linearly, you basically wouldn’t be able to sustain anything because of this “ideas are getting harder to find” idea or principle of the power law.
exponential vs superexponential
it seems that a key question which determines power concentration / unipolar vs multipolarity is whether “power” will accumulate exponentially or superexponentially. and if exponentially, whether various actors will have the same exponent or different exponents.
if it’s superexponential, then the power will accumulate to a single actor.
if it’s exponential and one actor continuously has a larger exponent than the other actors, then the power will accumulate to the actor with the larger exponent.
if it’s exponential and actors have the same exponents or the exponents are noisy over time but similar for all actors, then the current power distribution will be maintained.
This all begs a question of how to define power and how to track whether it’s exponential or superexponential. The easiest proxy for the near future is to look at the revenue growth of the companies or the gdp growth of the countries, depending on what level of resolution you are interested in. So far, we have fairly strong evidence that the revenue growth of the companies is exponential, and the same goes for the gdp growth of the countries.
Seems like companies have had dissimilar growth rates. Anthropic is growing at 10x/year, OpenAI is growing at 4-5x/year, etc.
Can you be ASI pilled and believe that we stay at 2% growth? probably not. If that’s the case then ASI really isn’t all that special.
It seems like the default path right now is that Anthropic becomes the unipolar power. If things change, and e.g. Anthropic and OpenAI both begin to have 7x / year growth, then they will both hold a roughly similar amount of power in the long run, barring other factors.
Other Factors
labs and countries might sabotage each other such that the tallest sunflowers get cut down a notch, which would bring down their growth rates to something closer to their competitors. alternatively, they might preemptively stop the peers who are behind from sabotaging them or catching up, such that even in an exponential growth environment, the first mover advantage will still be maintained.
on the other hand, labs and countries might cooperate with each other to prevent such negative-sum dynamics and instead coordinate such that even second movers are afforded a large ratio of power in the long run, out of generosity from the first movers.