The Self-Determining Eye

Of earrings, eyes, and legends

I have been thinking about the Whispering Earring a lot lately.

Scott Alexander’s short story describes a magical earring that gives its wearer perfect advice. The first thing it says is that the wearer would be better off taking it off. Nobody does. The advice that follows is excellent, the outcomes are excellent, and over time the wearer stops deliberating and simply executes what the earring suggests. By the end the instructions have descended from major life decisions to the level of muscle twitches. The wearer’s brain, on autopsy, is found to have atrophied. The story is short and it is doing a particular thing. It is not a warning about bad advice. It is a warning about good advice, delivered fluently and accepted because each acceptance produces a visibly better outcome than the alternative would have. The dependency is built out of small wins.

This is the standard parable people now reach for when they want to talk about cognitive offloading, and it works because it captures something the empirical literature is starting to confirm. Frequent users of large language models score lower on critical thinking measures. Students who write with AI assistance produce shallower arguments. The mechanism is not that the tool is bad. It is that the muscle you do not use weakens. The earring is the parable for what happens when a faculty of judgment is gradually replaced by something that produces, in the short run, better-looking results.

But the earring is a parable about a faculty we know we have. Judgment is something we can feel ourselves doing, or feel ourselves not doing. The earring’s wearer at least could have noticed, at the start, that they were no longer deliberating. The faculty had a felt presence and its absence had a felt absence.

There is a different problem that does not work this way. I want to talk about it through a different device, which I will call the self-determining eye.

Imagine a contact lens that improves your vision. Not by correcting refraction, which is what an ordinary lens does, but by interpreting. The lens identifies the objects in front of you, labels them, foregrounds the ones it judges important, and softens the ones it judges noise. It tells you, before you have consciously registered the scene, what the scene is. The improvement is real. You navigate faster. You miss fewer things that matter. Your accuracy on any test of seeing-with-the-lens goes up.

Now ask whether the wearer of this lens would notice their dependence. The earring’s wearer at least had the experience of being told what to do. The lens’s wearer has no such experience. The lens does not give them advice. It gives them a world. There is no felt moment of consultation, no gap between question and answer, because perception does not work that way. Vision is mostly unconscious inference. We do not see raw light; we see a world that the brain has already interpreted before the interpretation reaches awareness. Replace some of that inference with a model’s inference and the substitution does not feel like substitution. It feels like seeing.

This is the asymmetry. The earring takes over a faculty whose absence you could in principle notice. The eye takes over a faculty whose work was already invisible. By the time you have stopped seeing for yourself, there is no felt loss to register, because there was never a felt presence to begin with. The lens does not need to whisper. It does not need to warn. It does not need to ask. It simply arrives, and the world arrives with it, pre-interpreted.

The closest existing version of this device is not worn on the eye. It sits between a clinician and the patient.

Consider Legendary.

Legendary is a private company whose corporate slogan, until recently, was We control the market. Their software holds the chart, schedules the visit, routes the message, records the order, structures the note, assembles the bill, and carries much of the internal conversation of large American hospitals. In many of them, Legendary is not one system among others. It is the room in which the others appear.

Most accounts of the company, from clinicians and from journalists, treat the software as a documentation and billing system that has metastasized. Clinicians spend more time clicking than examining. Notes are bloated with templated language designed to satisfy billing codes. Inboxes fill with patient messages and lab results and pharmacy queries faster than anyone can read them. Alerts fire so often that clinicians learn to dismiss them without looking. The phrase “pajama time” entered the medical literature to name the hours physicians spend at home finishing their charts.

All of this is true. None of it is the most important thing about Legendary.

The most important thing about Legendary is that it determines what a clinician sees about a patient, in what order, with what foregrounding, before the clinician has formed an independent impression. Legendary decides what appears at the top of the chart and what is buried three clicks down. The problem list that summarizes who this patient is, clinically, is curated by a mixture of templates, structured coding, and natural language processing applied to prior notes. The medication list is reconciled by a workflow that privileges some sources over others. Risk scores for sepsis, for deterioration, for readmission, for fall risk, are calculated and surfaced by models running in the background, sometimes flagged to the clinician and sometimes shaping what the clinician attends to without ever appearing as an alert. Best-practice advisories and order sets nudge what gets ordered and what does not, often in ways that feel to the clinician like their own clinical reasoning.

This is not a complaint about the user interface. It is a description of what the system is. The software is the eye through which a substantial fraction of American medicine now sees its patients. The clinician’s gaze is real, and the clinician’s expertise is real, but the gaze arrives at a patient who has already been constituted by the system. The picture the clinician acts on is partly the system’s picture. And because the constitution happens in the substrate of perception rather than in the moment of decision, the clinician does not experience it as deferring to anyone. They experience it as seeing the patient.

The institutional version of this is the part the AI governance conversation has not caught up to. Hospitals running Legendary increasingly converge on a shared ontology of what a patient is. A locally meaningful distinction becomes an option in a dropdown. A messy social fact becomes a risk factor. A clinician’s suspicion becomes a coded problem or disappears into prose. Hospitals that once had distinct clinical cultures, distinct ways of thinking about a population, distinct local knowledge of what their patients tended to present with and how, now see through the same schema. The schema is not neutral. It was built by a private company, optimized for billing and operations, layered with predictive models whose performance the hospitals are not always permitted to independently validate, and deployed across institutions whose ability to evaluate it depends on capacities the deployment itself erodes.

A few years ago, independent researchers published an evaluation of Legendary’s proprietary sepsis prediction model, which by then was running in the background of patient care at hundreds of hospitals. They found the model performed substantially worse than the company’s marketing had claimed. It missed roughly two-thirds of sepsis cases at the threshold most hospitals used. It generated a high volume of false alerts. The company disputed elements of the methodology. The methodological dispute is not the point. By the time the dispute became public, the model was no longer just a thing to be evaluated. It had already entered the workflows, staffing assumptions, and habits of attention through which evaluation would have to occur.

This is what I mean by the self-determining eye, and what I mean by its loss. An institution with a self-determining eye is one that can ask, and answer, what it would conclude about a situation absent the tools through which it currently looks at that situation. It is one that can distinguish its own perception from the perception delivered to it. The hospitals running Legendary increasingly cannot do this, not because their clinicians are deficient, but because the infrastructure of perception is no longer a faculty the institution possesses. It is a service it consumes.

The AI governance conversation has been almost entirely about outputs. We argue about whether models are biased, whether they discriminate, whether their predictions are accurate, whether their decisions can be explained, whether their harms can be redressed. These are real questions and I do not want to dismiss them. But they all assume that the institution receiving the output is a stable perceiver evaluating a tool. The harder story, the one we are not telling, is what these systems do to the perceiving institution itself. A hospital is not the same hospital after a decade of seeing its patients through Legendary. An agency is not the same agency after a decade of seeing its mission through a dashboard. A regulator is not the same regulator after a decade of receiving its picture of an industry through analytics it does not control. The output frame cannot capture this, because the change is not in the outputs. It is in the eye.

This matters because institutions that have lost the self-determining eye cannot course-correct. The ability to notice that something is wrong, to form a picture of the situation that differs from the picture the system delivers, to ask what would be true if the tools were turned off, is the same muscle that the tools have replaced. The institution can still respond to inputs. It can still optimize against metrics. It can still produce outputs the system rates as good. What it cannot do is see the system itself. The eye that would have looked critically at the infrastructure is the eye the infrastructure has become.

The earring atrophied a faculty its wearer could feel. The eye reorganizes a faculty its wearer never thought of as a faculty at all. Somewhere in the difference is the part of this that is harder to write about, and harder to govern, and harder to walk back. The earring’s wearer dies smiling at the perfect last sentence in their head. The institution with the mediated eye does not die. It keeps working, keeps producing outputs, keeps meeting its metrics, keeps publishing its reports. It only loses the capacity to notice what it has stopped seeing.