The Solved World

What a relief!

Consider what it would mean for a parent to stop losing children to diseases that are currently treatable if caught in time but rarely caught in time. Consider what it would mean for a diabetic to stop doing the continuous arithmetic of staying alive, the carbohydrate counts and insulin ratios and blood sugar checks that constitute a second unpaid job running beneath every ordinary day. Consider what it would mean for a family to go through a year without the specific financial dread that comes from one person’s body behaving unexpectedly in a country where illness and bankruptcy share a calendar.

These are not abstract goods. They are the relief that most humans, most of the time, are actually hoping for when they think about a better future. If Artificial Superintelligence is what its most serious proponents claim, a system that would exceed human intelligence across every domain, then something like this relief becomes conceivable in structural terms. A system that genuinely understood human biology, psychology, and material need, and that genuinely wanted to reduce suffering, might be the first technology capable of delivering what every previous liberation technology only partially promised.

This essay is not going to argue that the dream is stupid. It is going to argue that the dream, followed honestly to its conclusion, leads somewhere its advocates have not fully examined. A superintelligence capable of comprehensively managing human life would not just solve our problems. It would quietly dissolve the conditions under which we develop as people. And this would not be a malfunction. It would be the system working exactly as intended.

-----

There is a distinction worth making before anything else.

Reducing suffering and comprehensively optimizing a life are not the same thing. Curing infectious disease, providing material security, extending lifespans or building institutions that work: these remove misery while leaving intact the conditions under which people form themselves through real decisions with real consequences. They change the parameters of a difficult world. They do not remove what you might call the resistance of reality, the fact that the world does not arrange itself for your benefit and that what you do matters.

What I mean by the resistance of reality is specific: the genuine possibility that your choices could fail in ways you did not choose and that no one else will fix for you. Whether a relationship will hold. Whether a career bet will pay off. Whether a decision you made a decade ago was worth what it cost. This kind of resistance is compatible with a great deal of comfort and security. It is not compatible with a superintelligence that has modeled optimal outcomes and is steering you toward them.

The objection to any argument like this is predictable: so you want people to suffer? No. The argument is narrower. Beyond a certain point, an intelligence working to optimize your life begins to remove the conditions under which you remain the author of it. Where that point is cannot be specified in advance, and that vagueness is part of what makes the problem hard.

-----

Think about what it actually means to be free.

Not free in the abstract, but free in the sense that matters to ordinary life: the sense in which you are the one making your choices rather than someone, or something, making them for you. This kind of freedom does not require poverty or hardship. It does require that the options in front of you are genuinely open, that you could actually choose wrong, and that you have some meaningful ability to push back if someone else is determining the terms.

A slave with a generous master who never punishes is still a slave. Not because of what the master does, but because of what the master could do. Her situation depends entirely on his goodwill. Her freedom exists at his discretion, not by any right she holds independently. The fact that he is generous does not change the structure. It is, in a way, the structure.

This is the problem with a perfectly benevolent superintelligence that manages human life comprehensively. Not that it would harm us. Not that its values would drift or its goals misalign. But that even if it acted with complete and genuine good faith, we would have no meaningful standing to contest its decisions. Our options would be what the system determined was optimal for us. The relationship would be one in which a vastly more capable will determined the field of our choices, and we navigated within it. This is a kind of captivity even if it feels like comfort, and it does not require any malice to be a problem.

This is why debates about AI alignment, about whether the system will do what we want, miss something important. Even a system that perfectly does what we want, in the sense of producing outcomes we would endorse, can constitute a relationship with us that is incompatible with genuine freedom. The question is not only whether the system is good. It is what kind of relationship having such a system makes us party to.

-----

There is a version of the objection that has more force than the suffering question.

Maybe the problem is not superintelligence itself but how it is deployed. Limit its reach. Let it handle medicine and logistics. Leave intact the domains where agency actually lives: relationships, creative work, moral choices, political life. Preserve, by design, the spaces where human beings still determine things for themselves. Nothing requires a superintelligence to manage everything. That what we are dealing with here is a governance choice and not an ontological destiny.

This objection is serious and deserves a serious reply.

The first problem is practical. A system capable enough to transform medicine, security, logistics, and material welfare is capable enough to have a more accurate model than any human institution of what constraints on its own deployment would actually produce. The argument for limiting the system requires the people doing the limiting to know better than the system what genuine human flourishing looks like. That might sometimes be true. But it asks a civilization that has struggled to govern far simpler technologies to design and maintain principled limits around something it cannot fully understand.

The second problem cuts deeper. “Preserve domains of human self-determination by design” has already conceded the central point. If human self-determination has to be deliberately preserved by an intelligence that has decided which domains of resistance are worth keeping and how much difficulty is good for us, then those domains are no longer places we inhabit freely. They are enclosures. Managed spaces within a managed world. The enclosures may be large and comfortable. They are still enclosures. And the being that designed them has already done the thing this essay is worried about: it has determined the conditions of our agency on our behalf.

-----

Neuroscience and psychology are worth bringing in briefly, not as proof, but as a check on optimistic intuitions about human resilience.

The human brain is organized around the experience of acting on the world and watching the world respond. Not passively processing information, but generating expectations and revising them when reality pushes back. Curiosity, motivation, and the sense of being a person who matters all depend on this loop. Research on what happens when people lose control over outcomes, when their actions stop making a difference, consistently shows that the erosion is not only psychological but physical, embedded in the brain’s motivational architecture. Passivity is not something people learn to embrace. What must be learned, through actual experience of consequence, is the expectation that actions matter. Take away the consequence and the expectation degrades.

This does not prove that a superintelligent manager would hollow out humanity’s inner life. The inference from controlled studies to civilizational outcomes is a long one and it should be acknowledged as such. But it does mean that the intuition behind most techno-optimism, that people are basically robust and will find meaning whatever the conditions, is not obviously correct. The relevant systems appear to require genuine engagement with a world that responds to what you do. They were not built for solvedness.

Notice what the enclosures concession actually describes. A world in which human agency is a managed amenity, something an intelligence has decided to preserve, in the domains it has decided matter, at the level of difficulty it has decided is appropriate, is not a world in which human agency exists. It is a world in which something that resembles human agency has been curated and maintained on our behalf. The difference between inhabiting your life and being granted a convincing version of it is not experiential. You might not feel it at all. But it is the difference the entire argument is about.

-----

A superintelligence that worked exactly as its advocates hope would not threaten us by going wrong. It would threaten the conditions of human agency by going right.

Curing disease, reducing poverty, extending lifespans, building institutions that function: these remain worth pursuing. The argument is not against relief. It is about what happens when the logic of optimization extends past relief into the management of how people encounter reality, make choices, and bear consequences. At that point the nature of the relationship changes. A will more capable than your own, acting on your behalf, determining what options are available to you, is a form of power over you regardless of how well it is exercised or how comfortable you feel inside it.

The question of whether a superintelligence can be made to want the right things is not the only question. There is also the question of whether what we want, understood fully, includes being the kind of creatures who determine things for ourselves in a world that does not defer to us. If it does, then the dream of liberation through a superior intelligence contains, inside its own logic, a form of the captivity it was meant to end.

That question needs to be worked through by people who are still doing the working. The act of sitting with it, following it where it goes, refusing the easy resolution, is itself a demonstration of what is at stake. It probably should not be outsourced.

The machine that kills us is one kind of catastrophe. This is another kind and it arrives with “relief”.