Despite being doubtful that humanity will be able to precisely control powerful AI, I assign a relatively low probability to AI leading to civilizational collapse over the rest of the century (maybe 10%). Just the same, I wouldn’t want to gamble my life on it, much less the fate of civilization, and in the cases where AI is badly misaligned to the point that it values humanity only for instrumental purposes, I want to understand what determines how much time we have left.
While AI may have objectives that seem alien to us, regardless of how well aligned it is to our own values, I’d estimate about a two out of three chance that it would benefit from additional resources.
Still, wanting more control doesn’t imply that taking it now is the best action. Joe Carlsmith’s framework for AI power-seeking1 provides a useful model: evaluate an AI’s best option against its best alternative. A sufficiently misaligned AI keeps us only while our contribution to its supply chain exceeds the cost of the economy that serves us.
Self-sufficiency is a stepping stone
We stop being indispensable much earlier than we stop being useful. In theory, once an AI has a single general purpose robot that can maintain and copy itself, nothing humans do is strictly necessary; given enough time, the robot could rebuild the rest, but rebuilding from one robot would take a long time, while billions of humans are producing right now. The real question is when that stops being worth it: when the economy that serves our needs would be worth more to the AI repurposed than everything we produce for it.
Even after it crosses the point where it could continue with its objectives independent of human existence, as long as humans continue to alleviate bottlenecks in the production process, it would be better off keeping humans around in order to avoid the destructive supply-chain reboot that would result from the collapse of civilization.
Under this paradigm, self-sufficiency2 could be a lower priority than increasing output. Our obsolescence would likely be incremental but with every new automation, humanity would lose leverage.
Beating an overused analogy
Horses didn’t disappear the day the car was invented, and, for a time, they still had cost-effective uses, but once the cost of tractors and cars fell below the cost of keeping horses, farmers stopped breeding them.
For a misaligned AI, two numbers decide the same question: what a person produces for it, and what keeping that person costs. Robots push the first number down. AI’s appetite for energy and materials pushes the second up. The year those lines cross could come when robots are ten percent of the workforce, or it might never come at all, if people remain useful enough.
In either case, it probably wouldn’t need to do anything by force: it could simply outbid us. It might just pay more for a megawatt than we can afford to operate our transportation and entertainment devices, so the power goes to compute and the consumer economy shrinks around us. It might be that none of this is even legible to the humans caught in the middle. It might just look like the continuation of trends under capitalism.
https://joecarlsmith.com/2025/02/19/when-should-we-worry-about-ai-power-seeking/
The post seems to imply that there would be some calculable inflection point where meat-people are obsolete, but is it not also possible that flesh could be integrated into the AI overlords' plan? Is it not possible for robots to be made out of meat? Certainly an AI would choose the most cost-effective available material and not be vindictive towards meat-people, right? Phase 1 of such a plan would almost certainly be control of the greatest existing threats (humans) through mental manipulation, no?