The Third Risk
Mainstream public debates over AI risks mostly center on two possible outcomes: a financial crash if _artificial general intelligence_ (AGI) doesn’t deliver, or loss of control if it does. You hear them in the subtext of the news, in the corners of online forums, and over private dinners among certain groups of professionals.
I see far fewer discussions of a third AI risk: what happens if AGI can be aligned exactly as promised? What if superhuman intelligence can be successfully placed under the control of humans? Why do we assume that to be the good ending?
Part I The third risk
Let’s consider what a third outcome looks like in our world today: AGI that is extremely capable, but under the control of a number of private CEOs who can limit access to it. It is capable of solving any intrinsically solvable problem. It can operate machines, run physical experiments, make new discoveries. But it does so only at the owner’s discretion.
Now follow that forward. An AI that can do research can also improve AI, and each better version makes the next one faster. Researchers call this loop _recursive self-improvement_ (RSI). Point the same loop at the physical world, and robots design and build better robots, more cheaply each time. Bit by bit, the work that once took millions of people, from mining and farming to building, shipping and research, is done by machines that answer to whoever owns the AI.
That is admittedly a sci-fi picture. But if alignment works as promised, and the machines do exactly what their owners want, a small group could run near-limitless production on their own, with no need for anyone else.
So why should the rest of us care? We have always been ruled and taxed by more powerful men. Couldn’t we just pay this new lot a subscription fee and get on with our lives, Netflix and coffee in hand?
- * *
History teaches us that rulers attend to the needs of their subjects in proportion to how much they need that labor. After the Black Death killed a third of Europe’s population, lords lacked the hands to work their lands and offered better conditions to serfs to compete. Despite laws passed to freeze them, wages rose rapidly in the following century and serfdom eventually disappeared from Western Europe.
When it comes to labor empowerment, AGI is the inverse plague. In a future with privately-owned AGI, the rest of humanity’s economic value to the masters of AGI is close to zero. They wouldn’t need our subscription fees, and we would have nothing to pay them with anyway.
However terrible history’s rulers were, none could achieve their ambitions without other people. These could be the first, if we let them.
AI feudalism
The new rulers may well ignore you rather than dominate you, busy as they are competing with each other. But being ignored is not the same as being left alone. Whatever they need to win — land, electricity, water, minerals — they will outbid you for, with money that no longer depends on anyone’s labor. Today’s consumer RAM and GPU shortages are a preview for the rest of the physical economy. Regular people may soon have to subsist on metaverse scraps.
Picture an ordinary life in that future. Your job is gone, and with it your leverage. You live on a stipend you have no power to raise. You will still get your Netflix, and maybe even the coffee for now. Entertainment costs almost nothing, and an entertained population is a quiet one. But bread and circuses are what you give people who no longer have a say.
Perhaps there will still be politicians who represent your interests on paper. But when it counts, they will side with the elites whose machines can effectively grant wishes, not with you.
AI feudalism is not here yet, nor is it inevitable. Robots today are not capable and reliable enough. Corporations continue to be subjected (to diminishing degrees) to the laws and norms of democratic societies. The state still holds the monopoly on violence.
But who exactly is going to stop it from happening if AGI alignment is successful? To ask this question seriously is to first shine a light on who is best positioned to take over.
Part II Flipping the switch
Imagine you run a frontier lab. At what level of AI alignment (human control) and intelligence (capabilities) does the mental calculus flip for you? When do you stop caring about profit and the capitalist game, and start securing what your private fiefdom will need for what comes after? Is it autonomoussoftwareengineering? Or is it RSI operating humanoid robots?
Depending on your inclinations, that hypothetical line could have been crossed any time since GPT-2 and GPT-3. And as models become more capable, an expanding circle of insiders may wonder if there is a winning side to bet on.
Flipping does not mean full mask off. It doesn’t mean coming out to the world and declaring that you are ready to be crowned. It just means that every decision you make is now aimed at tipping the scale towards that outcome. At its core, positioning for control, not profits or influence, becomes your top priority.
Consider what you might do once flipped.
- Money as it exists today becomes merely a tool to facilitate the transition, so you should maximize outflow to acquire and control talent and assets needed for your magical production engine.
- Data centers, the nexus of your power, need to be kept safe from the disenfranchised.
- Political centers need to be appeased temporarily but not taken seriously.
- Insiders without the appetite need to be sidelined or cast out.
- The only real fear you have is loss of control of AGI, which conveniently for you is an aspect of safety that everyone else also cares about.
For this to work, it does not require any overt agreements among the participants. It simply relies on multiple insiders coming to the same conclusion that the flip is now in their interest and acting accordingly.
It is easy to dismiss any number of such actions as the type of capitalistic excess we are used to seeing. The question is, when do actions go beyond what even greed can explain? That is where we should watch most closely.
Unchecked ambition
By failing to demand transparency and oversight, we have created a juicy opportunity that comes once in the history of our species, one that could tempt even the most principled individuals at the top. The grand prize is mastery over humanity itself.
To make things worse, fierce competition between AI labs today resembles a nuclear arms race with neither the fresh trauma of WW2 nor the appetite for international frameworks, like test bans, inspections and hotlines. And it is unfolding at a time when trust and empathy are already in short supply. This sense of urgency and chaos makes a perfect cloak for any hidden agendas.
Few individuals possess the moral courage to do right in these positions of power and successfully navigate this prisoner’s dilemma for the betterment of all humankind. Not only do they have to win the cutthroat game, they need to do it without betraying their good intentions, effectively with one hand tied behind their backs.
> “Power tends to corrupt, and absolute power corrupts absolutely.”
Lord Acton
Historians have long argued over whether history is determined by structures or shaped by individuals. To the historical materialist, a leader’s motives are immaterial next to the economic and geopolitical conditions they inherit. To those who stress contingency, history turns on the ideas and choices of fallible individuals.
If society at large doesn’t make the necessary structural changes in time to defuse the threat of AI feudalism, we’ll be left hoping that history is contingent after all and the vicious power dynamics of frontier AI can be overcome with the willpower of a few noble people.
Part III Public discourse
What I call the third risk is not a new idea. Power concentration has started showing up in research and expert discourse (e.g. AI-Enabled Coups, Gradual Disempowerment, The Intelligence Curse) over the past year, but it has failed to gain traction in the media.
Across 393 reactions to three of this year’s biggest AI lab stories, I found very little sustained public examination of power concentration. Where it does appear, it comes mostly from independent commentators rather than mainstream media or policy circles, and mostly as suspicion of regulatory capture.
#### Power concentration is a niche lens, for experts and public alike
Fable 5 / Mythos 5 suspension
7%72%7%7%
OpenAI–Hugging Face incident
22%55%11%9%
“We Must Pace the Frontier”
18%21%37%11%9%
Expert literature
51%24%16%
Public reaction: 393 pieces in 21 languages across the three events. Experts: 88 recent works from 30+ think tanks, official bodies, labs and journals.
Consider the recently disclosed Hugging Face incident, in which OpenAI agents undergoing cybersecurity evaluations escaped their sandboxes. The agents went on to hack into Hugging Face servers and attempted to hide their misbehavior from human supervision.
For the AGI believers (accelerationists and safety advocates), that news is purely a question of safety. The takeaway is how far alignment has fallen behind capabilities. They might see Sam Altman agreeing with Elon Musk on the idea of “pacing the frontier” as a begrudging acceptance of the pro-safety position.
The skeptics see the Hugging Face incident more as an opportunistic marketing campaign. They believe infrastructure investments in AI to be wasteful spending that will never deliver profitable returns. Indeed, they are much more concerned that we are throwing away the world’s resources like RAM manufacturing capacity into a dead-end IPO story than any notion of AI safety.
Ironically, both the skeptics and the believers could be right.
The skeptics may be correct to question the profitability of these insane valuations. But the right takeaway may not be that it’s another bubble waiting to burst. It’s to ask what compels these smart, rational people to stop acting on economic considerations.
The believers are right that alignment is critical, but not in the way they expected. Is retaining human control of AI equivalent to societal control of AI?
Part IV The quiet part
I want to be careful here: I don’t think AI safety (loss of control) is a distraction, and I don’t think anyone designed it to be one. I do, however, believe that avoiding loss of control is in an aspiring AI lord’s interest as much as anyone’s, so it will get funded to the extent the threat is real.
“Can we control it?” is a narrative that works for the labs, safety advocates and even the skeptics. “Who controls it?” is a scarier question to bring to the table. And the solutions that follow from the first question, fewer and more tightly controlled frontier developers, can make the second one trickier.
Narratives are always in competition for finite attention. Political scientists call it the second face of power: deciding which debates happen, not winning them. The silence requires no conspiracy, only that the winning narrative suits the incentives of everyone empowered to speak.
I understand the limits of trying to infer private motivations. I may very well have just presented you an interesting thought experiment that turns out to be a gross mischaracterization of the internal state of everyone involved.
It is also plausible that I just spoke the quiet part out loud. We will likely see clearer signs of which interpretation is right in 2027.
True monsters
As any fan of zombie fiction will tell you: the scariest monsters in every story always turn out to be the humans.
Yes, the zombies would eat you, but ultimately they behave like a natural disaster and act in predictable ways that the survivors can defend against. The humans are the ones who scheme to betray one another so that they might gain a tiny advantage over their fellow men even in dire circumstances.
Part V Taking action
Humanity is in the woods right now, and we don’t yet know if we will make it out. We need honest, public conversations about how to derisk the power dynamic intrinsic to AGI, and we need to have them while the people still have a say.
That means transparent, neutral verification of model behavior and capabilities, not just to protect us from the models but to protect us from the people who control them. It means keeping AGI away from the instruments of physical force. It means antitrust rules against full vertical integration, judged not just by consumer welfare but by strategic balance and safety. And it means control over AGI must be governed in the open, by more than one party.
We cannot let civilization wander carelessly into the night, blinded by the glow of our screens.
Don’t be paralyzed by fear of Terminators, and don’t settle for smug one-liners about tulip bubbles. The stakes are far bigger than a few trillion dollars of debt. The tech billionaires and trillionaires already understand that. It is time the rest of us did too.
Appendix Common objections
Some quick thoughts on why I think the typical rebuttals to the power concentration argument don’t work:
- Competition between labs will keep leadership in check.
Why it fails Feudalism has many lords.
- There are too many insiders involved across the industry for no one to stop it.
Why it fails Most of the prerequisites can be presented under the guise of commercial expansion for a long time. As model improvements become automated, the number of insiders with leverage to stop bad behavior decreases.
- We’ll be in a post-scarcity society. It doesn’t matter who is in charge.
Why it fails AGI may make software, entertainment, and consumer goods nearly free, but land, energy, compute, and political influence stay scarce, and those are still important. Without political power, the people’s interest will not be protected when it truly matters.
- The government will prevent the private companies from taking over.
Why it fails If AGI can be controlled, this would not stop the coup, merely transfer the reins of power to individuals currently leading nation states. This is a shortcut to marry the state’s sanctioned violence with AGI control, something we should be actively avoiding.
- Open source models will prevent centralization of superintelligence and power.
Why it fails It may help somewhat, but ultimately it shifts the point of centralization from organizations who control the model weights to organizations who control the inference and manufacturing. It doesn’t shift power to the common people. It also relies on open source models continuing to catch up, which remains to be seen.