.arunim.fyiStatus: very-rough draft
This tweet from Aidan McLaughlin, circa September 2024, feels pretty representative of the vibe at the time.
asi feels closer than agi if that makes sense
And the hype only intensified after we saw o3-preview.
My rough model at the time was “oh okay they’re just going to RLVR all the way up to ASI”, but it seems like that hasn’t turned out to be the case. More and more compute has been applied to the RL part of training, and while the returns have been large, they don’t seem to be large enough to directly get us to completely automating AI R&D.
I think I’m getting a bit off track with this line of thought, though. The actual thing that I wanted to discuss or write about here is different models for take-off or recursive self-improvement, and to actually examine how many of them, or which of them, are realistic given the evidence that we have accumulated so far. What I meant to convey with the RLVR point was that there was some prevailing feeling, or maybe it wasn’t prevailing, but there was some feeling at the time in September 2024, that the path to ASI was indeed clear and it would simply be to scale up RL training. I think that is just not the case now for a variety of reasons. There are still a variety of approaches that could be plausible for recursive self-improvement, and I’d like to examine what they are because, in my view, the risks that we have to consider from AI development are quite path dependent on how exactly are we getting to ASI.
What is RSI?? I think it just means humans out of the loop AI development. But like … how much? it could be the case that humans are cheaper than automated alternatives for a large variety of physical labor for some time. and still, we might attribute >90% of advances in AI R&D to AI.
Pure-software
We hold inputs roughly fixed, or capped under some humanly-achievable threshold, and AI of various forms is able to improve the efficiency of all the different parts of the chain that