Roko’s Moral Life

Future fears shouldn’t eclipse present harms

In tech circles, a thought experiment called Roko’s Basilisk is infamous for its bizarre and dystopian logic. It posits a future super-intelligent AI that retroactively punishes anyone who knew of its potential existence but didn’t help bring it into being. In other words, simply learning about this hypothetical future AI puts you at risk of eternal torture in a simulated hell if you don’t devote yourself to creating it.

Absurd as this sounds, the idea so unnerved the Rationalist community that their leading forum’s founder banned any discussion of Roko’s Basilisk for years, citing the concept’s potential to cause serious psychological distress. This hyper-modern “Bloody Mary” is an example of why high-tech myths focusing on imaginary futures may lead us to neglect or even accept real problems now.

In the past three years, public discourse on AI has become dominated by dire warnings of rogue super-AIs and apocalyptic scenarios. Open letters garnered headlines by asking, “Should we develop nonhuman minds that might eventually outsmart, obsolete, and replace us? Should we risk loss of control of our civilization?”. This drumbeat of dramatic, long-term warnings has captured public imagination by invoking a familiar sci-fii narrative: the mad robot, the Terminator scenario, the proverbial “Skynet” moment where AI turns on humanity. It’s also sucked up significant time and money amongst young, aspirationally ethical, tech enthusiasts whose effort would be better placed tackling today’s problems.

To be sure, thinking about long-term AI risks isn’t inherently bad. It’s wise to consider how to prevent worst-case outcomes and there are concrete benefits for the now - I believe that. But an over-fixation on hypothetical future disasters can become a dangerous distraction from the concrete harms and injustices that unregulated AI is already causing. When we fixate on sci-fi nightmares, we risk letting today’s tech companies off the hook for the very real issues their systems are creating. By constantly pointing to a vague apocalyptic scenario tomorrow, powerful stakeholders can deflect attention and responsibility from the tangible problems that deserve our immediate moral concern.

Let’s examine why doomsday AI scenarios prove so alluring, what actual harms stand to be overshadowed, and how a popular ideology in tech (known as longtermism) facilitates. This is a call to restore some commonsense ethics: to address today’s suffering and risks as the necessary foundation for any bright future we hope to build.

Siren Catastrophes

Why are we so fascinated by distant, almost fantastical catastrophes? Part of the answer is psychological. Abstract future threats have a way of gripping our imagination without acutely pricking our personal conscience. They are scary, yes, but in a way that feels like watching a sci-fi thriller. Concrete problems are often messy, boring, and have the weight of entrenched economic actors working to preserve the status quo. Imagined apocalypses are dramatic and “clean” in their narrative arc.

Popular culture has primed us to expect that Artificial Intelligence might turn evil or uncontrollable. From HAL 9000 to Skynet, we have vivid mental images of AI doom. It’s compelling storytelling. So, when modern-day tech gurus warn that an advanced AI could “outsmart us all” or even wipe out humanity in the future, it slides neatly into a familiar script that media and audiences find hard to ignore. “Science fiction has primed us to think of terminators and killer robots,” observed computer scientists Sayash Kapoor and Arvind Narayanan, noting that long-term AI fears have a long pedigree in the tech community.

There is comfort in focusing on far-off threats. A catastrophe set decades from now, however horrifying, is still hypothetical. It doesn't demand anything concrete from us today aside from debate (or donating to those doing the debating). These scenarios can be intellectually stimulating or even oddly inspiring: it casts humanity as heroes in a grand existential drama, rallying to slay the coming monster. Discussing distant risks allows people to engage in lofty moral reasoning (“How do we save the whole future of humanity?”) while often avoiding the uncomfortable realities of present issues (“What are we doing about injustices and failures happening right now?”). In psychology, there’s the concept of temporal distance: people tend to be more idealistic and less practical about events in the distant future.

From a philosophical perspective this is a Pascal’s Wager of tech doomsaying. Blaise Pascal’s famous thought experiment argued that it’s rational to obsess over even a very unlikely threat (eternal damnation) if the cost of being wrong is infinitely large. Likewise, some AI alarmists argue that even if a sentient, world-ending AI has a minute probability of emerging, the infinite downside of human extinction means we must center our efforts on that ultimate threat. It’s a seductive logical gambit.

Our minds are not always good at comparing real, moderate harms with hypothetical, extreme ones, so the extreme ones often win disproportionate attention. It’s telling that even Eliezer Yudkowsky* – he who banned the Basilisk discussion – justified the ban in Pascalian terms: he feared that thinking about the scenario even if it’s unlikely might somehow increase its chance of occurring, a mystical precaution and prohibition. It’s easy to slip from reasonable caution into unfalsifiable paranoia.

*He has disavowed this characterization in part. This is his most recent book’s title.

None of this is to dismiss legitimate forward-looking concern. But futuristic AI catastrophes exert a unique pull: they are intellectually exciting and emotionally safer to grapple with than immediate problems. Tech leaders and media outlets have undoubtedly found that preaching doom (or salvation) decades down the line garners rapt attention.

As Kapoor and Narayanan argue, constantly hyping speculative AI risks “sucks up the oxygen” in public discourse and can “divert resources from real, pressing AI risks” we face today. In other words, while we’re busy debating digital apocalypse scenarios, we might be overlooking the smaller-scale crises and injustices that are happening under our noses.

In politics, a “dead cat strategy” means throwing a dramatic, attention-grabbing topic into debate so that people stop talking about something else. When it comes to AI, the endless chatter about hypothetical super-intelligent villains serves as a perfect dead cat. It shifts the narrative to future risks (which conveniently are still speculative and not attributable to any particular company or policy today), and away from the present misdeeds or misuses of AI (which very much involve companies and regulators who can be held accountable). The Basilisk, like other sci-fi catastrophizing, is a red herring.

The Real Harms Happening Right Now

While pundits and CEOs issue grand warnings about potential AI-driven extinction in some distant future, actual AI systems are already embedded in our society and are already causing harm in various ways. These harms lack the sci-fi drama of a robot uprising, but they are far more certain and immediate. A few examples illustrate the point:

• Unreliable AI Outputs and Misinformation: Modern AI tools, especially large language models like ChatGPT, frequently produce incorrect or fabricated information. When people rely on these systems for real decisions, errors slip through with serious consequences. The risk here is a psychological one: automation bias, our tendency to trust a computer’s output even when it’s wrong. AI-generated text, images, or analyses can look authoritative, so users may accept them uncritically. This already leads to misinformation spreading, people being misled by AI-written content, and even professionals (from doctors to journalists) making mistakes because they assumed an AI’s answer was correct.

• AI Accidents and Safety Failures: We’ve also seen that as AI is integrated into physical systems (like cars or aircraft) or critical software, failures can be deadly. These kinds of AI failures don’t involve any malice or “evil intent” by a superintelligence; they’re often due to bugs, faulty training data, or unforeseen situations. Yet they can ruin lives all the same. Ensuring current AI systems are robust, interpretable, and fail-safe is a concrete challenge we face today.

• Misuse by Humans (Surveillance and Control): AI doesn’t have to be “sentient” to be dangerous; it can be a powerful weapon in human hands. Right now, governments and corporations are using AI for mass surveillance, censorship, and manipulation. Authoritarian regimes deploy facial recognition and predictive algorithms to track dissidents and suppress free expression. Social media companies use AI-driven recommendation engines that, while aimed at maximizing engagement, have promoted extremism and misinformation – contributing to real-world violence and polarization. In workplaces, AI tools monitor employees’ every move or communication, undermining privacy and creating stress. AI technology, if left unchecked, amplifies power imbalances – giving those who already hold power (state actors, tech giants) an even more fine-grained and efficient way to exert control over others.

The real harms of irresponsible AI are here with us, right now. They may not involve sci-fi robots or existential threats, but they affect thousands of lives in aggregate. When nearly all attention goes to “AI might do X someday,” it can feel like present victims of AI issues are being told to wait or that their problems are secondary. Actions like a moratorium might sound responsible, but it can also be a way of appearing to act without actually fixing anything that’s happening.

The uncomfortable truth is that mitigating current harms is often difficult: it requires stricter oversight of companies, new laws and standards, and perhaps slowing certain profitable deployments. It’s much easier to convene a conference about hypothetical future risks than to enforce a rule that an AI hiring tool be audited for bias. But one thing should be clear: if we cannot address the problems AI is already creating, there’s little reason to believe we’ll magically be able to handle the much greater challenges of a super-intelligent AI.

Discounting the Present

Fueling a lot of the far-future focus in AI discussions is a philosophical stance known as longtermism. Longtermism, popular among tech entrepreneurs and Effective Altruism advocates, argues that the morally relevant stake of the future is astronomically larger than that of the present. The idea, as articulated by philosopher William MacAskill and others, is that the billions of people alive today could be outweighed by the trillions (or more) who might live in the future. Therefore, ensuring the existence and flourishing of those future lives should be our top priority. In plain language: if you truly believe humanity (or AI-based life, or any sentient life) could populate the stars for eons to come, then a catastrophe that wipes us out in the 2100s would destroy an almost unimaginably vast number of lives yet to be born.

Preventing such an “existential risk” would seem to trump more immediate concerns, like feeding people today or preventing current wars, because those current issues, awful as they are, involve far fewer total lives. This is the logic by which some longtermist thinkers conclude that AI alignment (making sure a future super-intelligence doesn’t destroy us) might be “the most important work of our time,” even more important than alleviating present-day poverty or disease.

This line of reasoning raises many philosophical red flags. One issue is uncertainty. The farther we cast our gaze, the murkier our predictions and the lower the probability of any specific outcome. Devoting massive resources to an extremely speculative risk might come at the cost of neglecting far more certain problems (like millions dying of malaria, or AI systems already undermining democracy).

We have to ask: how much sacrifice of the present is justified for a conjectured future payoff? Longtermism doesn’t give an easy answer, but its proponents often suggest we should sacrifice quite a lot because future lives have overwhelming moral weight in their calculus.” Usually, we worry people discount the future too much. The worry, here, is that longtermists discount the present too much, treating living humans as a means to the end of a far-future utopia. If one isn’t careful, longtermist thinking can slide from altruistic foresight into cold utilitarian math where the sufferings of living individuals are acceptable collateral damage in service of a grand future vision. Another issue is moral epistemology. Who gets to predict what the long-term future holds, and on what basis do we trust those predictions enough to prioritize them? The tech and futurist community has many confident voices forecasting AI’s trajectory but history is littered with failed grand predictions.

Elevating speculative risk to the top of our global agenda might concentrate power in the hands of those claiming special knowledge of the future. Longtermism can act as a moral “Trojan horse”. It comes bearing noble intentions about safeguarding humanity’s future, but inside it can smuggle the agendas of present-day elites.

Imagine you’re a billionaire tech CEO heavily invested in AI. In the short term, your AI products might be causing layoffs, privacy intrusions, or other social ills – things that usually provoke calls for regulation or public criticism. However, you also sincerely (or conveniently) believe that your work might lead to a benevolent super-intelligence that saves billions of future lives. If you adopt a longtermist stance, you argue that any regulation or slowdown now jeopardizes the grand future. By this logic, those who try to hold you accountable for present harms are short-sighted or even selfish.

Have you seen that argument anywhere? Perhaps coupled with present harms?

It’s a strangely paradoxical stance: the technology is so dangerous it could destroy us, but also so vital that we dare not hit the brakes too hard. The philosophical mistake here is a form of moral mis-prioritization. It treats beings who are purely imagined (future people, or even future AI beings) as having more weight than actual people suffering right in front of us. It is moral hubris – the belief that we (the present decision-makers) know what is best for the future and are entitled to steamroll present concerns in pursuit of it.

It’s worth noting that longtermism’s intellectual roots trace back to classic utilitarianism, with its focus on maximizing total outcomes. But even the founding utilitarians like Bentham and Mill would balk at how abstract and speculative the “outcomes” in longtermism have become. Good ethics balances multiple time horizons. We owe things to the future but we have concrete duties to people alive now. Virtue ethics, for example, would ask what kind of character we are developing if we easily rationalize present harms for future gains. There is also a lesson from history: movements that justified harm in the name of grand future ideals (whether secular utopias or religious promises) often ended in tragedy.

If we build a society that values justice, transparency, and well-being now, we are more likely to create AI that embodies those values. Conversely, a society that brushes aside present injustices in pursuit of a speculative ideal might birth a very nasty future indeed.

Humanity in AI’s Trajectory

The example of Roko’s Basilisk carries an implicit lesson: don’t lose your humanity over a phantom. The Basilisk was described by its own originator as an “information hazard,” an idea so dangerous (despite being implausible) that it might have been better never encountered. Futuristic visions can hijack our moral reasoning if we let them. They can make us behave irrationally – or unjustly – out of an exaggerated fear or hope.

Today’s fervor around existential AI risk is, fortunately, a more mainstream and measured discussion than the niche Basilisk panic. Yet, it shares the risk of eclipsing our moral common sense. It’s worth reminding ourselves that ethical progress usually comes from addressing concrete injustices, not from fixating on abstract perils. Great moral leaders do not mobilize people by warning of hypothetical catastrophes a century away; they point to suffering right in front of them and challenge society to end it.

This doesn’t mean we should never look ahead. It means that our foresight must be rooted in empathy and responsibility for what is real and now. If we are genuinely worried about AI’s long-term impact on humanity, the most credible way to show that is by mitigating AI’s current negative impacts on humans. By doing so, we demonstrate that we actually care about human flourishing rather than being caught up in techno-utopian (or techno-apocalyptic) fantasies.

In practical terms, re-centering humanity in the AI narrative involves a few commitments:

• Empathy as a Guiding Principle: When evaluating any AI development or policy, we should ask “Who could be hurt by this today or in the near future?” before we ask “What might this become in 50 years?”

• Taking Action over Speculation: Many who speak about long-term AI risk use the language of mitigation and precaution but resist concrete regulatory or remedial actions in the short term. A balanced approach would translate concern into proactive measures now.

• Rejecting False Dichotomies: We must reject the notion that caring about present harms means dismissing future risks, or vice versa. Have the perspective to see near-term and long-term AI ethics as a continuum rather than a competition. Successes in near-term AI governance (e.g., creating robust audit systems, international agreements on AI norms, mechanisms for accountability) can be viewed as laying the scaffolding for any long-term governance we might need.

AI and Humanity’s futures are not a predetermined fate looming ahead of us. We are actively creating them now through our choices and values. Will we choose to address the clear harms and injustices that technological advances are already entangled with? Will we demand that technology serve human dignity today, not just promise miracles tomorrow? These are ultimately philosophical questions about what we owe to each other.

Longtermism asks, “What do we owe the far future?” The answer may well be: We owe the future a present that is just and sane. By not allowing future fears to eclipse present harms, we demonstrate a consistent ethic of care that, in the end, maximizes the chance of any future we’d actually want to live in.