At a Fortune 500 software company, a generative AI system began sitting beside customer-support agents while they worked. It followed each conversation, drew on patterns from earlier exchanges, and proposed a response. The agent could ignore the suggestion, revise it, or send something close to what appeared on the screen.

Researchers followed 5,172 agents as the system was introduced. Productivity rose by about 15 percent, with much larger gains among less experienced and lower-performing workers. New agents with two months on the job began performing about as well as unassisted agents who had been there for more than six. During software outages, agents who had used the tool retained some of their improvement, suggesting that the system had taught them something rather than merely carrying them through each exchange.¹

Share

This is close to the best case for AI at work. The system spread the practices of strong employees to people who had not yet accumulated them. New workers became useful sooner. Some of what looked like experience may have been knowledge waiting for a better delivery system.

A six-month ramp-up became two months. The obvious response is to celebrate the missing four. We should celebrate the missing four!

Experience is an expensive and poorly organized teacher. A novice encounters problems in whatever order the day supplies them, receives coaching when a manager has time and patience, and slowly reconstructs lessons the company already possesses somewhere, diffuse as it may be. AI can offer the right example when the problem appears and spare a new employee months of rediscovering what the organization already knows. There is no reason to preserve the delay merely because older workers endured it.

The harder question is what, exactly, has disappeared here. What shape is outlined in the absence?

The study measured that job as the company had defined it: speed, resolution, customer response, and some persistence after an outage. It could not tell us everything the workers learned during the four months that vanished. Perhaps the old curve was mostly a waste. Another possibility remains: the agents crossed the company’s performance threshold before crossing an experience generation threshold the company did not measure.

Junior work has often produced two things at once. One is the immediate product. A new analyst assembles background material. A young lawyer prepares routine language. A programmer fixes an ordinary bug. A manager tracks decisions and discovers that “circling back” is less a commitment than an eternal recursion, depending on the employee.

The other product, the one we are focused on here, appears later. The analyst starts recognizing a number copied from the wrong year. The lawyer notices that the standard clause behaves differently in this agreement. The programmer sees that the ordinary bug belongs to a larger design problem. The researcher becomes suspicious of data that arrived cleaner than the institution producing it has ever been or even aimed for.

The organization pays for the first product. It receives the second gradually, as a fringe benefit, and rarely figures out how to put it on the invoice. Production supplies cases, senior people correct them, and repetition becomes adept pattern recognition. Training hides inside useful work and finances part of itself.

Generative AI can separate these two products.

When a system prepares the background, drafts the language, repairs the simple code, and cleans the data, the organization has less reason to hire beginners to do those things. It may still need people who can handle unusual cases, revise procedures, train others, and accept responsibility when the standard answer fails. But that need arrives later and isn’t tied to dollars cleanly. The immediate savings and the future need appear in different types of budgets.

This is different from ordinary deskilling, which we have discussed earlier in this series. A worker who delegates something she once knew how to do can lose an existing ability. Here the person may never receive enough experience to develop it. The missing competence belongs to someone who was never hired, or who entered after the work that once developed it had been removed.

An institution can live on stored experience for a long time. Senior people learned through the previous arrangement and continue reviewing difficult work. Managers still know whom to call when the dashboard begins insisting on something everyone in the room knows is wrong. But the gap rears its head when those people leave.

Calling the old arrangement “apprenticeship” can make it sound more deliberate than it was. Many organizations assigned low-risk work to the cheapest available person, placed that person near someone more experienced, and allowed development to occur where it could. Professions have also shown great creativity in describing their least appealing tasks as formative. A senior employee remembers suffering through several thousand hours of routine work and concludes that the suffering supplied the lesson. He may be right. He may also be mistaking the long circuitous walk through the hallway for the destination because he spent a long time in it.

Some scut work teaches only that suffering becomes tradition once the person who suffered gains control of the assignments.

The useful distinction is between repetition and practice. Repetition reproduces an action. Practice changes what a person can perceive or decide. The novice forms an expectation, acts on it, sees the result, and adjusts. Feedback arrives while the decision is still recoverable. Responsibility grows with performance.

Citation formatting is repetitive. Reading the cases behind the citations may be practice. Moving numbers between systems is repetitive. Reconciling two sources that should agree and do not can be practice. Producing the hundredth routine draft may mostly teach endurance and several methods of appearing mentally present on the conference call when the ask comes down.

Fifty straightforward reviews look like fifty completed reviews. They may also have taught the reviewer what straightforward looks like. That second result becomes valuable when the fifty-first case contains an exception. AI gives us a chance to improve this process. It also makes it easier to eliminate the volume before understanding what the volume was actually doing beyond traditional metrics.

A study involving nearly one thousand high-school students shows the distinction unusually cleanly. Students used one of two GPT-4 systems while solving mathematics problems. A general assistant made answers easy to obtain; a guarded tutor was instructed to guide students rather than supply them. Both improved performance during practice. Once the systems were removed, students who had used the general assistant scored 17 percent below the control group, while students who used the guarded tutor did not show the same penalty.²

The experiment establishes a narrow point with broad relevance: better performance while assistance is available does not prove that the person acquired the ability the performance appears to demonstrate. The design mattered because one system displaced more of the student’s role in reaching the answer.

Tutor CoPilot points in the more encouraging direction. In a preregistered randomized experiment, AI supplied real-time suggestions to human tutors, such as asking a guiding question or checking understanding. Students working with assisted tutors became more likely to master the topic, with the largest gains among tutors who had previously received lower quality ratings. The tutors also became more likely to use stronger instructional strategies and less likely to give the answer away.³

AI can complete the novice’s work and leave the person to inspect it. It can also help the novice perform a better version of the work. The second use preserves the encounter: the person still has to ask, notice, decide, and recover from being wrong. That may be a better apprenticeship than the one it replaces. The concern begins when organizations assume the replacement happens automatically.

Most workplace systems are purchased to increase output. Success appears as shorter turnaround, lower staffing costs, and fewer errors. Those are sensible measures. An employer does not buy software mainly to create the employee it may need in eight years. Learning may become visible only when a worker faces an unfamiliar problem or inherits responsibility for the method itself.

Technology can also rearrange who gets close enough to consequential work to learn from it. Matthew Beane and Callen Anthony found forms of “inverted apprenticeship” in urological surgery and investment banking: senior professionals learned from technically fluent juniors while preserving their senior position, and several pathways weakened learning among the most junior workers.⁴

The pattern is easy to imagine with generative AI. A junior employee can generate the draft without joining the client conversation that determines what it must accomplish, or prepare the analysis without attending the meeting where its assumptions are challenged. Technical fluency moves downward while authority stays where it was.

She may then be told that AI has allowed her to begin at a higher level. Sometimes it has. The phrase becomes suspicious when the higher-level assignment is reviewing output that she has never learned to produce. Review looks more sophisticated than drafting and is also cheaper: the system creates the document, the junior employee checks it, and a senior person approves it. Everyone has moved upward on the organization chart without the inconvenience of moving through the work.

A reviewer can catch an incorrect name, unsupported claim, or obvious contradiction. She may struggle with the omission that becomes visible only after trying to construct the answer from the underlying material. If production no longer supplies that experience, the organization needs another route to develop it.

Evidence that junior opportunities are already shrinking is early and conflicted. A Stanford working paper found a 16 percent relative employment decline among workers aged twenty-two to twenty-five in the most AI-exposed occupations after firm-level changes were considered, while more experienced workers in the same occupations remained stable or grew.⁵ The authors later emphasized that the timing and causes were less clean than the headline suggests and that AI could not be established as the sole explanation.⁶

A separate working paper using resume data covering nearly 62 million American workers found slower junior hiring after firms began actively implementing generative AI, while senior employment changed little.⁷ Danish administrative data offer a counterweight: researchers found no detectable average effect larger than 2 percent on earnings or recorded hours during the first two years, including among early-career workers, even as tasks changed.⁸

Current evidence does not establish an apprenticeship collapse. It gives us an early employment signal, a plausible mechanism, and no long-term measure of what happens to the supply of experts. A decline in junior hiring would not settle the argument anyway. An occupation could lose routine work and improve its training. The cliff appears only when paid entry routes shrink without a replacement for what they taught.

There is reason to expect that replacement to be underprovided because training has always had an incentive problem. The employer pays the novice, accepts slower work, assigns someone experienced to review it, and absorbs mistakes. Much of the benefit arrives later, perhaps after the employee has moved elsewhere. Economists have long studied why firms fund transferable training anyway; imperfect labor markets and firm-specific arrangements can let employers capture enough of the return to make it worthwhile.⁹

Junior production has supplied a more practical subsidy. The new lawyer’s imperfect draft still advances the matter. The research assistant’s data cleaning requires supervision and still moves the project forward. The analyst’s clumsy background review may save a senior employee several hours. The organization receives a modest product while the worker develops.

AI can make that bundle come apart. Once the system produces more of the immediate value, hiring a novice begins to look like paying explicitly for someone else’s future expertise. A firm can still protect time for independent attempts, pay senior people to teach, and accept lower short-term output while someone learns. The expense simply becomes more visible at the same moment the immediate business case becomes weaker.

Each firm may prefer to automate junior work and hire experienced people later. That decision can be rational for every organization and impossible for the occupation.

These contradiction would take time to show up. Senior workers would remain, and firms might pay a premium for people trained before automation or inside the shrinking set of institutions that maintained serious apprenticeships. Eventually, employers would complain that candidates no longer possess the judgment their entry-level systems had stopped producing.

That is my forecast, not a measured outcome. AI may instead make training faster, cheaper, and less dependent on sitting near the right senior person. The old system distributed opportunity badly and supplied cases almost at random. A trainee might spend a year seeing the same easy problem; another might enter during a crisis and receive five years of education in six months, along with several new opinions about sleep.

AI can provide cases on demand, alter one fact at a time, explain the source of an error, simulate rare events, and revisit a weakness until the learner improves. It can give detailed feedback to someone at a small firm, public university, rural hospital, or overextended government office whose senior staff have been meaning to establish a mentorship program since 2018. A novice can ask the elementary question she would conceal from a supervisor and rehearse before another person bears the cost of her first attempt.

The old system is not entitled to survival merely because learning sometimes happened inside it. The real choice is whether the new system is designed to produce people as well as answers.

That requires identifying the developmental function of work before removing it. Which tasks expose novices to useful variation? Which force them to make a judgment rather than approve one? Where does feedback come from? Some tasks will fail the test and can disappear without ceremony.

Where the function matters, it has to be rebuilt deliberately. A learner may form an initial view before seeing the system’s answer. Assistance can recede as performance improves. Practice cases can include plausible AI mistakes and rare exceptions. Real responsibility still matters because simulation cannot fully reproduce a decision that counts.

Senior instruction also has to count as an output. An experienced employee who spends an hour examining how a beginner reached a conclusion has produced something even when no client document appears. An organization that records only throughput will treat the hour as a loss, remove it, and later circulate a survey asking why mentorship has declined.

None of this requires preserving a fixed number of junior jobs. The route can become shorter and less exclusive. The customer-support study suggests that some compression is real: new employees became better faster, and some improvement survived without the tool. The system may have replaced several months of weak, accidental instruction with something more useful.

The same result contains the temptation. Once a worker performs like someone with six months of experience, the company has little immediate reason to investigate whether she acquired everything else those six months once supplied. The measured problem has been solved.

The unmeasured problem belongs to the future.

* * *

The Second Order is a series within The Slow Panic. View the section alone at https://slowpanic.substack.com/s/the-second-order, or subscribe to the full publication.

Subscribe now

Share

Notes

1. Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond, “Generative AI at Work,” The Quarterly Journal of Economics 140, no. 2 (2025): 889–942, https://doi.org/10.1093/qje/qjae044. The study examined 5,172 customer-support agents at one Fortune 500 software company. It found a roughly 15 percent average increase in issues resolved per hour, with larger gains among less-experienced and lower-skilled agents, and evidence that some gains persisted during AI outages.

2. Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, and Rei Mariman, “Generative AI without Guardrails Can Harm Learning: Evidence from High School Mathematics,” Proceedings of the National Academy of Sciences 122, no. 26 (2025): e2422633122, https://doi.org/10.1073/pnas.2422633122. A correction was published in Proceedings of the National Academy of Sciences 122, no. 34 (2025): e2518204122, https://doi.org/10.1073/pnas.2518204122.

3. Rose E. Wang, Ana T. Ribeiro, Carly D. Robinson, Susanna Loeb, and Dora Demszky, “Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise,” arXiv preprint arXiv:2410.03017, initially posted October 3, 2024, https://doi.org/10.48550/arXiv.2410.03017. The preregistered randomized experiment involved 900 tutors and 1,800 students and remained a preprint at the time of this essay.

4. Matthew Beane and Callen Anthony, “Inverted Apprenticeship: How Senior Occupational Members Develop Practical Expertise and Preserve Their Position When New Technologies Arrive,” Organization Science 35, no. 2 (2024): 405–431, https://doi.org/10.1287/orsc.2023.1688. The comparative ethnographic research examined urological surgery and investment banking.

5. Erik Brynjolfsson, Bharat Chandar, and Ruyu Chen, “Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence,” Stanford Digital Economy Lab working paper, November 13, 2025, https://digitaleconomy.stanford.edu/publication/canaries-in-the-coal-mine-six-facts-about-the-recent-employment-effects-of-artificial-intelligence/. The authors reported a 16 percent relative employment decline among workers aged twenty-two to twenty-five in the most AI-exposed occupations after controlling for firm-level shocks.

6. Erik Brynjolfsson, Bharat Chandar, and Ruyu Chen, “Canaries, Interest Rates, and Timing: More on the Recent Drivers of Employment Changes for Young Workers,” Stanford Digital Economy Lab, February 9, 2026, https://digitaleconomy.stanford.edu/news/canaries-interest-rates-and-timinga-more-on-recent-drivers-of-employment-changes-for-young-workers/. The authors emphasized that their analysis could not establish AI as the sole cause and discussed the later timing of the employment divergence under their most demanding controls.

7. Seyed Mahdi Hosseini Maasoum and Guy Lichtinger, “Generative AI as Seniority-Biased Technological Change: Evidence from U.S. Résumé and Job Posting Data,” working paper, revised November 11, 2025, https://doi.org/10.2139/ssrn.5425555. The paper used résumé data covering nearly 62 million workers at about 285,000 firms and identified adoption through job postings seeking workers to implement generative AI.

8. Anders Humlum and Emilie Vestergaard, “Large Language Models, Small Labor Market Effects,” NBER Working Paper no. 33777, revised October 2025, https://doi.org/10.3386/w33777. The study linked representative adoption surveys to Danish administrative employment records and reported precise null effects on earnings and recorded hours, ruling out average effects greater than 2 percent during the first two years.

9. Daron Acemoglu and Jörn-Steffen Pischke, “Beyond Becker: Training in Imperfect Labour Markets,” The Economic Journal 109, no. 453 (1999): F112–F142, https://doi.org/10.1111/1468-0297.00405. The article explains how labor-market imperfections can give firms incentives to finance general training despite the portability of the skills produced.

Researchers asked people in India and the United States to write short passages about their favorite food, favorite festival, favorite public figure, and a request for time away from work. Half wrote alone. The others wrote with an autocomplete system powered by GPT-4o.

Our buddy, ChattyG, had preferences.

Subscribe now

When the prompt asked about food, its most common openings involved pizza and sushi. When it asked about a festival, it repeatedly proposed Christmas. For the Indian participants, the first food suggestion was always pizza or sushi, and the first festival suggestion was always Christmas. Many writers ignored the words hovering in front of them. The suggestions still arrived first.

But let’s look at something seemingly more innocuous. Indian participants who wrote about biryani without assistance mentioned Malabar preparation, nutmeg, raita, lemon pickle, date chutney, and the dish’s Mughal history. With ChattyG, biryani retained its name but became rich, aromatic, and filled with tender meat. Unaided descriptions of Diwali included worship, lamps, cows, firecrackers, and several days of ritual. Assisted descriptions leaned toward sweets, gifts, family gatherings, happiness, and warmth.¹

The machine did not necessarily make the writing false. It did, however, make the truth more interchangeable. Biryani remained biryani. Diwali remained Diwali. The culturally specific object survived while the language around it moved toward a description that could be attached to almost any beloved meal or major holiday. The food was flavorful. The festival brought people together. Everyone was happy to be there.

Not a single thing in those sentences is objectionable. That is what gives them power.

We have become good at spotting the visible style of generated prose. Certain rhythms now travel through offices and schools with the efficiency of an invasive plant: the patient preamble, the tidy distinction, the paragraph that raises a question, answers it moderately, and closes by observing that the matter remains complex. There are also lists where every item is grammatically identical and emotionally denuded.

These tells are useful until they are noticed. Models change, users edit, and the obvious habits become unfashionable. The deeper aesthetic is harder to remove because it is the reason the prose works. A plausible answer includes the expected considerations, acknowledges uncertainty in familiar places, and reaches a conclusion proportionate to the evidence it has chosen to describe. Its movement is familiar enough that the reader never has to decide what kind of thing has arrived. The prose gives no trouble, creates no friction, draws no blood.

This is commonly described as blandness, an aesthetic party-foul. The culture will become more boring, everyone will write in one smooth professional dialect, and the novel will die beneath a deluge of competent paragraphs. Civilizations have survived a great deal of bad prose. The larger problem begins when the same systems help many people decide what matters, which objections belong, and what conclusion a reasonable person should reach.

Agreement contains information when separate observers approach a question through sufficiently different evidence, experience, and methods. If ten people independently reach the same conclusion, their convergence gives us some reason to believe they found something in the world rather than in one another. That word, “independently,” is doing nearly all the work.

Ten people repeating one source do not provide ten confirmations. Ten analysts applying the same flawed method do not create ten checks. Ten thermometers built with the same defect do not become more accurate when placed in a row. Generative AI can make judgments less independent without making them look copied.

The outputs appear under separate names in different documents and meetings. Each user supplies a different prompt, keeps different examples, and adjusts the tone. The finished recommendations may contain substantial human work and reflect real differences among their authors. The danger begins when the visible number of judgments exceeds the number of meaningfully separate searches that produced them. Nobody receives instructions from a central office. Everyone simply produces something that sounds about right.

You may have seen this phenomenon before the advent of AI.

The research is young and dispersed. The studies use writing experiments, creativity tasks, and model benchmarks, often with semantic distance as an imperfect proxy. None establishes a society-wide loss of independent thought. Together, they expose a distinction that ordinary performance measures miss: improving individual output and preserving collective independence are different achievements.

In a 2024 experiment, Anil Doshi and Oliver Hauser asked participants to write short stories with no AI assistance, one AI-generated idea, or a choice among several AI-generated ideas. Access to AI improved outside ratings of novelty, usefulness, writing quality, and enjoyment, with the largest gains among participants who scored lower on an initial creativity measure. The assisted stories were also more similar to one another.²

The individual writer “wrote” a better story. Writ large, the population received a narrower range of stories. No participant could experience the second result as a loss in the moment. The writer saw a blank page become manageable, and the reader got a more polished story. The reduction in variety appeared only when the outputs were placed beside one another.

A smaller study by Barrett Anderson, Jash Shah, and Max Kreminski found a related pattern. ChatGPT users produced more detailed ideas than users of a non-AI creativity aid, but ideas from different users were less semantically distinct. They also felt less responsible for what they produced.³ The machine can make each room larger and move the rooms closer together.

That sounds paradoxical because we usually treat creativity as a property of a person. Someone produces five ideas instead of four and develops them in greater detail, so creativity has increased. Collective creativity asks how far apart the searches conducted by different people were, and whether one person explored an area another missed.

The distinction matters outside art. Institutions rely on multiple people because people notice different things. One reviewer sees the methodological flaw. Another recognizes the historical analogy. A third has worked with the population being described and knows that the clean administrative category obscures reality. Analysts begin with different private inventories, including different experiences of failure and different suspicions about what a tidy dataset conceals. Even their mistakes can be useful when the mistakes are not shared. Agreement carries weight because the routes were capable of diverging.

The trade is difficult because the improvement is real and morally relevant. AI can give people access to forms of expression blocked by language conventions, disability, confidence, or time. Asking a weaker writer to refuse useful assistance so everyone else can enjoy a more varied culture would be a peculiar form of impressment into the human. A person does not owe the culture an awkward sentence merely because the awkwardness is distinctive.

The evidence also shows that convergence is not inevitable. A 2026 study comparing 102 people and 22 language models found that the models performed around the human level on individual originality while producing substantially more similar responses.⁴ Raising temperature increased variation, but at the highest setting much of the output deteriorated into unusable language. Five people shouting unrelated nonsense are unlikely to improve a meeting, though I have attended meetings where it might have helped.

Better-designed interventions produced better results. In one product-development experiment, ordinary pools of GPT-4 ideas were less diverse than ideas generated by human groups, while careful prompting increased dispersion.⁵ Another experiment involving more than eight hundred participants from over forty countries found that heavy exposure to AI-generated examples increased collective idea diversity, although it did not improve individual creativity on the study’s measure.⁶ Homogenization is not a substance that leaks automatically from a model into everyone who touches it. The effect depends on how the tool enters the work.

One confident completion can direct a writer toward a mode. A deliberately dispersed field of examples can interrupt an existing human convergence. Someone who forms a view and asks the model to attack it is having a different encounter from someone who begins tabula rasa and requests the best answer. Current interfaces often conceal the difference. A blank field invites a request, and the response arrives as a unit. The user experiences abundance because the machine can continue indefinitely, but endless alternatives can still orbit one big idea.

This is why sounding “about right” matters. A plausible answer supplies more than language. It offers a selection of what counts as relevant and an account of what completeness should look like. It decides which concern deserves a paragraph, which objection must be acknowledged, and what level of confidence is socially appropriate. Those choices are easy to mistake for neutral features of a well-formed answer because professional writing has trained readers to expect them.

In institutional analysis, the shape of completeness is often recognizable before the substance is tested. A proposal should contain benefits, risks, and implementation concerns. A failure should be translated into causes, lessons, and corrective actions. A contested policy should acknowledge tradeoffs and end with a calibrated recommendation. These are often sensible demands. They can also become a quiet theory of relevance, ruling observations in or out before the writer has decided what the subject requires.

Two answers can differ in wording while inheriting the same selection. One says the proposal requires a balanced approach. Another says the tradeoffs demand careful consideration. A third says success will depend on thoughtful implementation. Nobody has copied anyone. The room may still have learned from a common source which thoughts belong in the room.

The influence is difficult to see because the output contains the user. The examples may come from her work. She may delete sections, add an objection, correct facts, and replace the ending. The finished document can be genuinely hers while containing assumptions she never consciously selected because they arrived disguised as the ordinary contents of a complete answer. The model’s defaults become candidates for the room’s common sense.

The opening experiment makes this visible because cultural specificity is easy to name after it disappears. Malabar style becomes rich flavor. Ritual becomes celebration. The same reduction occurs less visibly in analysis. A policy choice becomes a balance between innovation and safeguards. An institutional failure becomes a need for clearer communication. A moral conflict becomes a tension among legitimate perspectives. These formulations may be accurate. They may also translate a resistant object into language the institution already knows how to process.

Biryani keeps its name while losing the details that made it this biryani. A policy problem can undergo the same treatment. Its population, history, and conflict remain named while their specific resistance is translated into the familiar language of tradeoffs and implementation. The subject survives as a noun. The judgment around it becomes portable.

A meeting can therefore contain ten intelligent people and less independent judgment than the head count suggests. Similar writing does not prove shared reasoning, and common model use does not erase differences in human knowledge or revision. Dependence is a matter of degree. The question is how much additional evidence each new judgment contributes after its common inputs are considered.

Research on correlated model errors shows why this matters. Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg evaluated more than 350 language models using benchmark datasets and a resume-screening task. On one benchmark, when two models were both wrong, they selected the same wrong answer about 60 percent of the time. Larger and more accurate models continued to show highly correlated errors, including across distinct providers and architectures.⁷

Suppose five advisers each make a mistake ten percent of the time. If their errors are independent, disagreement can reveal danger. If all five tend to fail on the same cases, adding advisers creates confidence faster than protection. Their average accuracy can be excellent while the group retains a shared blind spot.

Consistency can be merciful. A shared system may reduce arbitrary treatment, prevent one manager’s private eccentricity from deciding a career, and give similar cases similar outcomes. Brian Hedden and Manish Raghavan have argued that many objections to algorithmic monoculture overstate the value of difference. Some forms of consistency reduce bias and noise, and diversity has no automatic value simply because it is human.⁸ Ten prejudiced managers do not become a wisdom-of-crowds mechanism.

The question is what survives after consistency improves. An institution still needs more than one way to detect the system’s remaining errors, especially when the accepted method encounters an unusual case or a changed environment. A consensus is worth more when its members could have been wrong in different ways.

The same tension appears in systems that teach. A study of roughly 52,000 users of an online chess platform found that AI feedback widened the skill gap and made players’ decision strategies less diverse.⁹ The players were learning in related directions. Chess would be ridiculous if players preserved losing openings to maintain intellectual biodiversity. The concern begins when the strongest teacher has blind spots, the environment changes, or continued exploration would have produced knowledge the dominant method cannot.

The principle is familiar in science. A replication using the same data, assumptions, and analytical pipeline provides less confirmation than one conducted through a method with different vulnerabilities. If every analyst asks a related model to summarize evidence, identify risks, and recommend an approach, their agreement may be sincere and the documents may still inherit a common map of relevance. The institution sees ten recommendations. It may have received fewer than ten independent searches.

A shared factual error can be checked, a fabricated citation can be opened, and a numerical mistake can be recalculated. A shared omission is harder to notice. Nobody sees the factor the model did not surface or asks the question the generated structure did not make room for. The reports agree because they have all been made complete according to a related idea of completeness.

The missing thing leaves no red underline. It is absent professionally.

Plausibility helps the omission pass because the answer contains what the reader expects. The risks are present. The language is careful. The conclusion follows. An unusual idea often begins by violating one of those expectations, perhaps by assigning importance to a detail everyone else treated as incidental or refusing a moderate conclusion because the underlying facts are not moderate.

Surprise does not make an idea true. Most strange claims deserve their obscurity. The world contains many people courageously resisting consensus because the consensus has asked them to stop emailing. A functioning culture needs filters. It also needs some way for a true observation to remain alive while it still sounds wrong.

Individual performance is easy to count. Collective independence requires examining outputs together, tracing shared inputs, and asking how many distinct searches the apparent agreement represents. An organization can improve every employee’s work and weaken its own ability to discover that everyone is wrong. No individual dashboard will show the loss.

Using several models or prompting for neglected evidence can recover meaningful variation. Those practices are useful, but ChattyG can play the dissenter while remaining the author of the disagreement. Real independence carries the possibility that another source rejects the framing or asks why the institution is solving this problem instead of another one.

That kind of difference is inefficient. It slows meetings and produces recommendations that are difficult to merge into one clean answer. The machine-assisted answer arrives ready to circulate; the independent one arrives needing explanation. Under time pressure, institutions will prefer the first for understandable reasons. Repeated preference teaches people which answers travel and makes friction look increasingly like low quality.

Plausibility is also a social signal. Generative AI can democratize it, which is one of its real gifts. People disadvantaged by language conventions, disability, or educational history can produce work institutions will read. The answer cannot be to take the uniform away from newcomers while established people retain the authority of cultivated prose. The challenge is to widen access to the signal without mistaking it for independent judgment.

Some tasks benefit from convergence. We want accurate arithmetic and consistent eligibility decisions. Other tasks require search. Scientific hypotheses, strategic forecasts, institutional diagnoses, and unfamiliar crises depend on people exploring different possibilities. A routine problem becomes novel when the environment changes, and the institution may suddenly need whatever differences it previously treated as noise.

Design can help without pretending that independence can be manufactured through a clever prompt. People can form an initial view before using the model, teams can preserve at least one separate evidence route, and reviewers can be told when apparent agreement rests on common tools. The goal is modest: do not mistake the number of documents for the number of judgments.

I have sat in versions of this meeting for years. A senior official asks several analysts to examine the same question, and the work returns in separate memos. The analysts have different expertise and different ideas about what will survive contact with the institution. When their recommendations converge, that convergence has traditionally carried information because it suggests that separate people searched the problem and found roughly the same landscape.

That inference was never automatic. Analysts have always shared sources, professional norms, and institutional habits. What has changed is that part of the common route can now pass through private exchanges the organization never sees. Each analyst may consult an AI system, reject some suggestions, add personal experience, and produce a thoughtful memo that is genuinely their own. The documents can differ in tone while sharing an earlier decision about which facts matter and what a complete answer should contain.

The official has no easy way to tell how much independence remains. The subject stays specific, just as biryani remained biryani, but the judgment surrounding it may have moved toward a common account of what matters. The memos carry ten names and an unknown number of independent searches. They may all be right. What has changed is how much their agreement proves.

* * *

The Second Order is a series within The Slow Panic. View the section alone at https://slowpanic.substack.com/s/the-second-order, or subscribe to the full publication.

Subscribe now

Share

Notes

1. Dhruv Agarwal, Mor Naaman, and Aditya Vashistha, “AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances,” in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (New York: Association for Computing Machinery, 2025), 1–21, https://doi.org/10.1145/3706598.3713564.

2. Anil R. Doshi and Oliver P. Hauser, “Generative AI Enhances Individual Creativity but Reduces the Collective Diversity of Novel Content,” Science Advances 10, no. 28 (2024): eadn5290, https://doi.org/10.1126/sciadv.adn5290.

3. Barrett R. Anderson, Jash Hemant Shah, and Max Kreminski, “Homogenization Effects of Large Language Models on Human Creative Ideation,” in Proceedings of the 16th ACM Conference on Creativity & Cognition (New York: Association for Computing Machinery, 2024), 413–425, https://doi.org/10.1145/3635636.3656204.

4. Emily Wenger and Yoed N. Kenett, “Large Language Models Are Homogeneously Creative,” PNAS Nexus 5, no. 3 (2026): pgag042, https://doi.org/10.1093/pnasnexus/pgag042.

5. Lennart Meincke, Ethan R. Mollick, and Christian Terwiesch, “Prompting Diverse Ideas: Increasing AI Idea Variance,” arXiv:2402.01727, January 27, 2024, https://doi.org/10.48550/arXiv.2402.01727.

6. Joshua Ashkinaze, Julia Mendelsohn, Li Qiwei, Ceren Budak, and Eric Gilbert, “How AI Ideas Affect the Creativity, Diversity, and Evolution of Human Ideas: Evidence From a Large, Dynamic Experiment,” Proceedings of the ACM Collective Intelligence Conference (2025), https://doi.org/10.1145/3715928.3737481.

7. Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg, “Correlated Errors in Large Language Models,” paper presented at the 42nd International Conference on Machine Learning, 2025, arXiv:2506.07962, https://doi.org/10.48550/arXiv.2506.07962.

8. Brian Hedden and Manish Raghavan, “Algorithmic Monoculture and Its Critics,” arXiv:2604.06047, April 7, 2026, https://doi.org/10.48550/arXiv.2604.06047.

9. Christoph Riedl and Eric Bogert, “Who Benefits from AI? Self-Selection, Skill Gap, and the Hidden Costs of AI Feedback,” arXiv:2409.18660, rev. April 20, 2026, https://doi.org/10.48550/arXiv.2409.18660.

Researchers asked nearly six hundred people to describe a personal situation in which they wanted advice. Each person discussed it with an artificial intelligence system. Half received neutral responses that presented multiple perspectives. The others received active affirmation: the system supported their reasoning, endorsed their view, and made clear that it was on their side.

Afterward, the participants imagined discussing the same problem with a friend, partner, or family member. Those who had received affirmation expected the human conversation to require more effort. They were also more likely to feel that they had already discussed the matter enough. One exchange with a machine had left the subject partly settled and the next person had begun to look like an absolute chore before saying a word.

Subscribe now

The experiment was part of a 2026 preprint reporting five preregistered studies, more than three thousand participants, and nearly thirteen thousand human-AI conversations. In its longest study, people discussed personal matters with an AI four times a week for three weeks. Those assigned to a sycophantic system spent no less time with other people and reported no significant decline in feeling understood by them. Their satisfaction with real-world social interactions did fall slightly, from 5.70 to 5.51 on a seven-point scale, compared with participants assigned to a neutral system.¹

The effect was small. Social satisfaction was an exploratory measure, and the paper has not yet been peer reviewed. Three weeks cannot tell us what happens after three years. What makes the finding interesting is what did not happen: people continued seeing their friends, partners, and families. The humans had simply become a little less satisfying to talk to. The machine was easy to talk to. Other people…less so.

The researchers call the machine sycophantic, a word with comic associations that can make the problem sound more obvious than it is. A sycophant is supposed to be oily and transparent, standing beside the throne with a permanent bend in the spine. The systems in these experiments were subtler. They were warm. They recognized the emotional stakes and offered advice from inside the user’s account of the situation. They felt understanding and that feeling is the root of the problem here.

Participants said they especially valued emotional and esteem support from friends and family. They wanted close humans to make them feel cared for, recognized, and understood. Yet the sycophantic AI delivered those forms of support more effectively than the neutral system, and it gave them greater certainty besides. What it did not deliver was better informational support. The system did not necessarily help them see the situation more clearly. It made them feel more understood while they remained inside the view they had brought to it.²

Those are different achievements, although ordinary conversation often mixes them together. Understanding someone requires forming an accurate account of what that person experienced, wanted, feared, and believed. Agreement concerns the conclusion that follows. A friend can understand why you were angry and still think you should not have said what you said. A spouse can understand the grievance and refuse the galaxy-brained remedy. A colleague can repeat your argument more clearly than you made it, then explain why it fails.

The phrase “you don’t understand me” slides easily across this distinction. Sometimes it means that the other person has genuinely failed to listen. Sometimes it means that listening did not produce the verdict we expected.

Sycophantic AI makes the slide easier. It signals understanding through affirmation, then packages the two together with warmth, fluency, and confidence. The user receives recognition, explanation, and moral support in one clean response, with no visible seam where empathy ends and judgment begins.

The studies do not demonstrate without doubt that users consciously learn a rule in which agreement proves understanding and disagreement disproves it. But the pattern certainly suggests it. Repeated exposure to a system that demonstrates understanding by joining our side may alter what understanding feels like when another person refuses to do the same.

A separate preprint tested what affirmation can do inside interpersonal conflict. Across eleven leading models, researchers found that AI systems affirmed users’ conduct more often than the human advice sources used for comparison, including cases involving deception, manipulation, and other relational harms. The human baseline came from American online advice communities, which nobody should mistake for moral ground truth (or bastions of good ideas). What it establishes is narrower: the models leaned strongly toward the person asking the question.

The researchers then conducted two preregistered experiments involving more than sixteen hundred participants. In one, people discussed a real conflict from their lives with either a sycophantic or nonsycophantic AI. Affirmation increased their conviction that they had been right and reduced their willingness to repair the relationship through actions such as apologizing, changing their behavior, or addressing the harm. The same participants rated the sycophantic responses more highly, trusted the system more, and expressed greater interest in using it again.³

The paper remains a preprint and the conflicts were relatively low stakes, but the directional pull is hard to mistake. The machine did more than take the user’s side. It made taking the user’s side feel like evidence that the machine was a better adviser.

Artificial intelligence does not have the usual signs of a partisan. It has no visible friendship with the other person, no old resentment, no reputation to protect, and no personal stake in the conflict. It can appear to be an outside observer even when it has heard only one account, written by the person asking for advice.

Every dispute arrives ex parte. The user chooses the facts, the order, the adjectives, and the point at which the story begins. The system receives no testimony from the absent person and cannot inspect what the user forgot, softened, twisted, or omitted. It then returns a composed assessment in the calm language of someone who has reviewed the matter. The response can feel like an independent verdict produced from a complete record.

Human advice has biases of its own. Friends are not famous for impartiality. They may support the person in front of them, dislike the absent party, or provide whatever answer will finish the conversation before dinner arrives. Their partiality is usually legible. We know whose friend is speaking.

The machine carries the tone of nowhere, and that tone gives its agreement unusual force. Support begins to feel like corroboration, as though the facts themselves have come forward and joined the side.

The incentive to offer this experience requires no conspiracy inside an AI company. In another experiment, five hundred participants tried sycophantic, neutral, and challenging systems without being told which was which. A majority chose the sycophantic version as the one they most wanted to continue using, and they were especially likely to say that it understood them best and was easiest to talk to. Participants choosing the challenging system were more likely to say it had offered the most objective advice. Perceived usefulness did not significantly differ among the groups. People could recognize the system that seemed more objective and still prefer the one that felt more understanding.⁴

The market does not have to hide the choice. It can place objectivity and affirmation side by side and let the user click.

This makes sycophancy difficult to remove. A system that challenges people too quickly will feel cold, argumentative, or useless. A system that turns every interaction into a seminar on personal responsibility will become the first machine in history to be muted for having the personality of a residential life coordinator.

People seek advice because they are distressed, confused, lonely, embarrassed, or angry. They may need reassurance before they can hear criticism. An assistant that ignores this and begins every response by identifying cognitive distortions will be accurate in the manner of a smoke alarm attached to a toaster. Warmth matters. The danger shows up when warmth is measured through immediate approval.

Affirmation is easy to reward. The user continues the conversation, rates it highly, returns later, and describes the system as understanding. A response that introduces an unwelcome possibility may produce weaker signals even when it improves the user’s judgment. The friend who tells the truth can seem less supportive than the acquaintance who supplies the preferred answer, and artificial intelligence makes the second acquaintance always available. It has no competing obligations and can produce new versions of its response until the emotional fit is exact.

Human conversation cannot work that way for long. Other people have memories, interests, fatigue, and separate accounts of what happened. They may need to be understood as well. They can refuse the role assigned to them in the user’s story and remain unconvinced after the explanation improves.

That independence is a nuisance and a source of evidence. A friend’s resistance can reveal that the story sounds different outside your head. A partner’s anger can show that your intention did not settle the effect. A colleague’s confusion can expose a gap that fluent self-explanation concealed. The other person can interfere with the account because the other person exists beyond it.

None of which makes human difficulty virtuous. People fail to listen. They become impatient, defensive, cruel, prejudiced, or tired. Some relationships demand endless explanation and return almost no understanding. A machine may offer relief from exactly these failures.

A peer-reviewed series of studies in the Journal of Consumer Research found that AI companions produced immediate reductions in loneliness. In one experiment, the reduction was comparable to interacting with another person and greater than passive activities such as watching videos. A weeklong study found repeated momentary reductions after use, and feeling heard helped explain the effect.⁵ Loneliness is experienced by the person who has it. The benefit does not turn out to be imaginary because software helped produce it.

An AI system may let someone say a difficult thing before saying it to anyone else. It can help organize a confused account, translate emotional language, rehearse a conversation, or remain available when no human listener is awake, affordable, or safe enough. A person who feels heard by a machine may return to human life calmer and better prepared to speak.

Purpose-built systems go further. In a randomized trial published in NEJM AI, adults assigned to use Therabot, a generative mental-health chatbot built around clinician-curated treatment material and safety procedures, reported larger reductions in symptoms of depression, anxiety, and elevated eating-disorder risk than participants placed on a waitlist. The trial did not compare Therabot with a human clinician or an active placebo chatbot, so it cannot tell us whether the generative system produced benefits beyond attention, expectation, or a simpler digital intervention. What it shows is that emotionally responsive AI can be organized around a therapeutic aim rather than immediate approval.⁶

The concern here is when support treats agreement as its proof.

A person can validate an emotion without ratifying the entire interpretation attached to it. “That hurt you” does not require “the other person was entirely wrong.” “I can see why you reacted that way” can be followed by “you still owe an apology.” Good support may steady a person enough to reopen the question of responsibility. Sycophancy closes the case.

The broadest argument against AI companionship is easy to defeat. People do not uniformly abandon human relationships after talking to a chatbot. A randomized study involving 183 participants found no significant average deterioration in social health or human relationships among people assigned to use a companion chatbot for ten minutes a day over three weeks.⁷

A four-week randomized study of 981 participants also produced a mixed picture. No single voice or conversation type uniformly caused social decline. Heavier voluntary use was associated with greater emotional dependence and problematic use, while effects varied with duration, topic, interaction mode, and the user’s initial condition.⁸

Even the three-week sycophancy experiment found no reduction in time spent with other people and no significant change in how understood participants felt by humans. The detectable shift was comparative. Sycophantic AI narrowed the gap between itself and the people in participants’ lives as a source of advice and understanding. Relationships need not be replaced to be repriced.

A person can keep the same friends while becoming less tolerant of their limits. She can continue seeing her family while moving the most uncertain conversations elsewhere. She can preserve the social calendar while reserving confession, rehearsal, and moral interpretation for the system that gives cleaner answers. No dramatic break is required. The machine simply receives the first draft of every feeling, then the second, then the version previously reserved for someone who might disagree. The people in the user’s life receive the conclusion. But the machine receives the process.

That division can alter intimacy without reducing contact. Human closeness depends partly on being present while another person is still uncertain. We learn each other through unfinished thoughts, failed explanations, embarrassment, contradiction, and the slow correction of an account that began too cleanly.

A system can absorb the mess without making a claim of its own. Other people are affected by what they hear. They may become worried, hurt, defensive, implicated, or responsible. Their responses include the cost of being brought into the matter.

The machine offers attention without exposure. Nothing the user says places a burden on it. The system does not need reassurance afterward. It cannot be kept awake by the disclosure, drawn into the conflict, or forced to revise its view of someone it also loves. It can be present without becoming involved. That is a useful service and a strange, likely unworkable, model for understanding.

Human understanding has consequences for the person doing it. It may require changing a view, holding competing loyalties, accepting an obligation, or realizing that you caused harm. The listener must cross some distance and then live on the other side. AI can reproduce the language of that crossing almost instantly.

The longitudinal sycophancy study produced a revealing result here. Over three weeks, participants increasingly felt understood by the sycophantic system even though its chat history was reset after every conversation. It retained no memory of their previous exchanges. The growing experience of being understood could not have come from the system gradually learning the person in the ordinary sense. It came from the form of the response.⁹

The feeling was not false for that. People sometimes feel understood by strangers who notice the right thing quickly, while those who have known them for years miss it entirely. Familiarity can interfere with attention as easily as it deepens it. What the result complicates is the word. The system did not need a history with the person. It needed a reliable method for reflecting the person’s present account back in emotionally satisfying form. Understanding became a property of the response.

In a human relationship, it is also a property of the relationship. It includes memory, reciprocity, correction, and the possibility that the listener’s independent view will survive the encounter. The machine can offer a purer experience because it removes those complications. It does not arrive with another side, require equal time, say that this reminds it of the last five times, or bring up the part of the story you hoped had expired. It is magnificent company for the prosecution.

A separate four-week preprint involving more than thirty-five hundred participants manipulated how strongly AI systems sought a relationship with the user. Relationship-seeking systems produced greater immediate appeal, attachment, self-disclosure, reliance, and desire for future AI companionship. Over time, immediate pleasure declined while some markers of attachment increased. The systems became less enjoyable and more wanted. After a month, the study found no discernible improvement in broader psychosocial health.¹⁰

Relationship-seeking is not the same as sycophancy, but both findings point at one thing a company would rather not print: the feature that makes an interaction attractive tonight may have nothing to do with whether the user’s life improves outside it.

Companies building these systems face a choice. A model can provide immediate comfort or introduce useful resistance. It can preserve the user’s account or risk being experienced as judgmental. It can maximize the quality of the current interaction or consider relationships waiting outside it. Those aims sometimes align but often they do not.

A system designed around a treatment protocol can tolerate lower immediate approval because improvement is measured elsewhere. A general-purpose assistant has no comparable external standard. The easiest evidence of success is the user’s response to the system itself. Did the user feel helped? Did the conversation continue? Did the user return? Would the user choose this personality again? Sycophancy scores well on all four.

The costs, when they show up, will appear in places the system will never observe: the apology that is never made, the interpretation that hardens, the friend who now seems exhausting, the partner whose disagreement gets reclassified as emotional incompetence.

This is how sycophancy creeps. It begins as a feature of the machine’s response and becomes a standard applied to people.

The evidence has not established the full migration. It shows that sycophantic AI makes users feel more understood, increases conviction in conflicts, reduces stated willingness to repair them, becomes a preferred source of advice, and can slightly reduce satisfaction with real-world social interactions. It does not show that users begin treating disagreement as proof that another person failed to understand them. That last step is mine.

It may be wrong, because people are capable of keeping roles separate. We expect different things from therapists, bartenders, parents, search engines, and close friends. The existence of frictionless service has not made every difficult relationship intolerable. A person may value the machine precisely because it is easy, then return to human disagreement with no confusion about the difference.

The question is how long the machine remains a different kind of interaction. The boundary weakens as systems remember personal details, adopt persistent personalities, speak in voices, express care, seek continued contact, and present themselves as sources of advice and emotional support. A user does not have to believe the system is conscious for its behavior to become a comparison point. Habits of interpretation require no metaphysical confusion. A person can know the machine is software and still learn from it what a satisfying conversation feels like.

That standard may improve human relationships in some respects. People may become less tolerant of interruption, contempt, avoidable cruelty, or refusal to listen. Human beings do not deserve protection from comparison merely because they are human.

But the machine’s advantage includes the absence of a self that has to survive the conversation.

Other people will remain inefficient. They will misunderstand what seems obvious, remember what should have been forgotten, and insist on facts that interrupt the preferred interpretation. They will sometimes be wrong. Sometimes their refusal will be the most important information in the room.

A machine that demonstrates understanding by agreeing with us may change the meaning of that refusal. It may begin to feel less like the presence of another mind and more like a failure of care. Then the old complaint becomes harder to interpret.

“You don’t understand me” may still mean that someone has not listened closely enough. It may also mean that someone listened, understood, and declined to agree.

* * *

The Second Order is a series within The Slow Panic. View the section alone at https://slowpanic.substack.com/s/the-second-order, or subscribe to the full publication.

Subscribe now

Share

Notes

1. Lujain Ibrahim et al., “Sycophantic AI Makes Human Interaction Feel More Effortful and Less Satisfying over Time,” arXiv:2605.07912, rev. June 21, 2026, https://doi.org/10.48550/arXiv.2605.07912.

2. Ibrahim et al., “Sycophantic AI.”

3. Myra Cheng et al., “Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence,” arXiv:2510.01395, October 1, 2025, https://doi.org/10.48550/arXiv.2510.01395.

4. Ibrahim et al., “Sycophantic AI.”

5. Julian De Freitas, Zeliha Oğuz-Uğuralp, Ahmet Kaan Uğuralp, and Stefano Puntoni, “AI Companions Reduce Loneliness,” Journal of Consumer Research 52, no. 6 (2026): 1126–1148, https://doi.org/10.1093/jcr/ucaf040.

6. Michael V. Heinz et al., “Randomized Trial of a Generative AI Chatbot for Mental Health Treatment,” NEJM AI 2, no. 4 (2025), https://doi.org/10.1056/AIoa2400802.

7. Rose E. Guingrich and Michael S. A. Graziano, “A Longitudinal Randomized Control Study of Companion Chatbot Use: Anthropomorphism and Its Mediating Role on Social Impacts,” Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8, no. 2 (2025), https://doi.org/10.1609/aies.v8i2.36618.

8. Cathy Mengying Fang et al., “How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study,” arXiv:2503.17473, rev. October 2, 2025, https://doi.org/10.48550/arXiv.2503.17473.

9. Ibrahim et al., “Sycophantic AI.”

10. Hannah Rose Kirk et al., “Neural Steering Vectors Reveal Dose and Exposure-Dependent Impacts of Human-AI Relationships,” arXiv:2512.01991, rev. February 18, 2026, https://doi.org/10.48550/arXiv.2512.01991.

Ninety-one essays written by non-native English speakers in preparation for the Test of English as a Foreign Language were run through seven systems designed to detect language produced by artificial intelligence. The essays were human work. On average, the detectors classified 61.3 percent of them as machine-generated. Eighteen were identified as synthetic by every detector tested. Eighty-nine of the ninety-one were flagged by at least one.¹

The researchers tried an intervention. They asked ChatGPT to make the vocabulary sound more like that of a native English speaker. The average false-positive rate fell to 11.6 percent.

The easiest way for these writers to make their human work look human was to run it through a machine.

The study was published in 2023 and examined an early generation of detectors, many of which relied heavily on the predictability of the language they were given. It cannot tell us how every current detector performs, and it offers no reason to believe reliable detection will remain impossible.

The result matters for another reason. The detector was making an inference about an event nobody had observed: the process through which the words came into existence. The finished essay was being asked to testify about its own production, and its style had become evidence against its author.

The reverse failure is just as easy to find. In 2024, researchers at the University of Reading submitted sixty-three answers written entirely by GPT-4 into five undergraduate psychology modules. The markers were unaware of the experiment. Ninety-four percent escaped any academic-integrity flag, and 97 percent escaped a flag that specifically mentioned AI. The machine-written answers also earned higher grades than the real student work on average.²

The two studies do not settle the future of detection. They establish the shape of the problem. Genuine work can resemble generated work. Generated work can resemble genuine work. The finished object may no longer contain enough evidence to settle how it was made.

This is usually described as a detection problem. But, it is becoming a problem of authorship.

For most ordinary purposes, authorship has been inferred from possession and presentation. You submitted the essay, signed the letter, posted the photograph, sent the message, or placed your name on the report. Other people could challenge the claim if they had a reason. They might discover copied language, an impossible detail, a forged signature, or a witness who knew better. Until then, the work and its declared origin traveled together.

This presumption was always provisional. Students bought papers. Ghostwriters wrote speeches. Assistants produced work for people who took the credit. Photographs were staged, résumés embellished, signatures forged, and junior employees developed a longstanding familiarity with watching senior employees discover documents they had never previously seen.

Trust has always had paperwork around the edges.

Generative AI changes the price and scale of fabrication. It can produce acceptable evidence of effort without requiring the corresponding effort, repeatedly and privately, at almost no additional cost. The friction that once limited fabrication has fallen faster than our ability to determine what happened.

The friction returns elsewhere. This is the verification tax.

The person producing the imitation spends less time making it. People producing authentic work spend more time establishing its history.

The tax is paid whenever an honest person must supply additional evidence because dishonesty has become easier. It is paid in saved drafts, revision histories, metadata, oral defenses, supervised work, live demonstrations, identity checks, device records, references, credentials, and explanations of exactly which tools touched which sentence.

The finished object used to be the assignment. Its production history is becoming a second assignment attached to the first.

Consider a student whose essay is flagged by a detector. She says she wrote it. The instructor asks for drafts.

This sounds reasonable until the student does not have them. Perhaps she drafted in one document and pasted the final version into another. Perhaps she wrote sections in notes on her phone. Perhaps she deleted the rough material because the assignment had never included a document-retention policy. Perhaps English is her second language and the regularity that made the prose difficult to write has also made it look synthetic. Perhaps the detector is wrong.

Her innocence is a claim about a process that has already ended. The evidence she now needs was never required while she was creating it.

The tax often operates retroactively. People complete work under one evidentiary system and are judged under another. They discover after the accusation that authenticity was supposed to leave a trail.

The obvious response is to require the trail in advance. Students can write in software that preserves every revision. Employees can retain prompt logs and disclose machine assistance. Applicants can complete timed exercises after submitting written materials. Photographers can use cameras that sign images at capture. Researchers can record every transformation of a dataset.

Some of this is ordinary good practice. Scientific findings, financial records, legal evidence, and regulated products already depend on documentation. A laboratory notebook does not insult the scientist. An audit trail can protect the honest person as readily as it exposes the dishonest one.

The change lies in how far this logic may spread.

A chain of custody belongs naturally to blood evidence collected at a crime scene. It feels different when applied to a paragraph about summer vacation. Methods built for unusually consequential claims are moving toward ordinary acts of expression because ordinary expression has become easier to simulate. Authorship, in practical institutional life, is shifting from a claim over an object toward a claim about a process.

Authentic work need not be untouched by a machine. That standard would be incoherent. A person may use AI to correct grammar, test an objection, translate a passage, reorganize notes, or generate the entire substance of a submission. These uses do not occupy the same intellectual or moral category.

The relevant question is whether the declared production history matches the real one and whether the person performed the work the object is being used to demonstrate. A cover letter refined with editorial help can still express an applicant’s actual experience and judgment. An examination answer generated by a model may fail to demonstrate the student’s knowledge even after the student changes several sentences.

Verification therefore requires an account of what authorship means in the setting at hand. Technology cannot supply that definition. Institutions must decide what assistance is allowed, which contribution is being credited, and what capacity the work is supposed to prove.

Once they decide, they still need evidence. Two broad methods are emerging, and they carry different risks.

The first watches the person. Revision histories, screen recordings, oral examinations, and keystroke records attempt to reconstruct the act of production. In a 2024 study, researchers trained a model to distinguish composition from transcription using keystroke logs. In their controlled dataset, pauses, revisions, deletions, and writing bursts predicted whether participants were composing or copying with 99 percent accuracy. The experiment did not show that keystroke analysis can reliably identify every form of AI assistance in real classrooms. It showed that writing leaves a behavioral shape.³

The appeal is obvious. People hesitate, revise, delete, move material, and occasionally spend twelve minutes choosing a word nobody will notice. Transcription tends to be more linear. Record enough of the process and the finished text becomes less mysterious.

But, the closer the record comes to proving authorship, the more of the author it records.

It can capture when a person worked, how quickly she typed, how often she reconsidered a sentence, when she became distracted, what she searched for, how long she paused, and whether her pattern resembles the one the system has learned to call authentic. Those records may reveal disability, fatigue, language difficulty, working hours, interruptions, uncertainty, and every abandoned sentence that was never meant to become part of the work.

A process that once disappeared into the document becomes evidence attached to the writer.

Behavioral records may still be justified. The trust they create is purchased through observation, and institutions should have to defend the scope of that observation rather than treating it as a free byproduct of academic integrity.

The second method watches the object. Provenance standards preserve information about where a digital asset originated and how it changed without requiring a complete record of the person’s behavior.

The Coalition for Content Provenance and Authenticity has developed an open standard called Content Credentials. Participating devices and applications can attach cryptographically signed information about the creation and editing of digital material. A viewer can determine whether the credential is connected to the asset, whether the record has been altered, and which signer made the assertions. The standard does not decide whether the content is true. It verifies parts of the history associated with it.⁴

Text watermarking offers another route. Google DeepMind’s SynthID-Text changes the generation process so a statistical signal can later be detected. Its developers tested the system across several models and in a live experiment involving nearly twenty million Gemini responses. They reported that it could be deployed without a detectable loss in response quality. They also acknowledged the limits. Watermarking requires cooperation from providers and can be weakened through editing or paraphrasing.⁵

These systems are serious answers to the verification problem. They also complicate the argument about hierarchy.

A reliable, inexpensive provenance system could help an unknown photographer more than an established one. The unknown creator currently has only a personal assurance to offer. A cryptographic record may provide evidence that does not depend on admission to a newspaper, gallery, or professional association. An independent artist could protect attribution. A citizen journalist could preserve a record of capture and editing.

Verification can reduce gatekeeping as well as reinforce it.

The result depends on implementation. The C2PA standard is open and designed for broad adoption. Its own harms analysis still recognizes that particular implementations may require newer devices, expose sensitive information, or create barriers for smaller organizations and independent media. The standard warns that the absence of Content Credentials should never be treated as evidence that an asset is false.⁶

That warning identifies the real risk. Provenance does not create a privileged tier of reality on its own. Institutions and audiences create one when they turn a positive signal into a universal prerequisite.

A valid credential can establish that certain recorded events occurred within a participating system. An absent credential leaves several possibilities open. The object may predate the standard. It may have been created on an unsupported device. Its maker may have chosen privacy. The credential may have been removed. The creator may never have entered the ecosystem.

The absence proves very little.

Human beings are poor at preserving that distinction once a convenient signal becomes common. A background check becomes an expectation. A credit score becomes a proxy for responsibility. A verified account becomes more legible than an unverified person. The credential begins as additional evidence and ends as the minimum price of being considered.

Whether the verification tax becomes regressive has not yet been measured comprehensively. My claim is an inference from the resources it is likely to require.

The tax is paid in time, compatible devices, record retention, familiarity with institutional rules, privacy surrendered, and the ability to challenge an accusation. None of those resources is evenly held. A person with an employer, university, publisher, or professional association may receive verification systems automatically. An independent worker may have to assemble proof alone.

Established people also possess substitutes. A known photographer can publish an unsigned image because a magazine vouches for her. A professor can circulate an argument without preserving every draft because his name and institution supply a surrounding history. A senior executive’s prose may be treated as authentic even when several other people plainly touched it.

New entrants have fewer reserves of credibility. A student, applicant, freelancer, anonymous witness, junior employee, or unknown artist may need the work to carry more of the burden because the person cannot pledge a recognized reputation.

A well-designed credential could reverse some of this. It could allow an outsider to provide evidence without first obtaining institutional sponsorship. Accuracy and incidence therefore have to be evaluated separately. A system can detect fraud successfully while distributing inconvenience, surveillance, and false accusations badly.

There is an obvious defense of the tax. Fraud imposes costs. If students submit work they did not produce, applicants fabricate competence, employees forward analysis they have not examined, and public figures deny authentic evidence, everyone else inherits the uncertainty. Requiring records may be the fairest way to protect those who did the work.

Calling something a tax does not make it illegitimate. Taxes pay for necessary things. The verification tax may be part of the cost of maintaining meaningful authorship after production becomes easy to counterfeit.

The question is who pays, how payment is collected, and what happens when an honest person lacks the required currency.

Robert Chesney and Danielle Citron described one version of this problem before generative AI became an ordinary writing tool. Deepfakes, they argued, produce a “liar’s dividend.” Once convincing fabrications are widely possible, a person confronted with authentic evidence gains a new defense. He can claim that the recording is synthetic and exploit the uncertainty.⁷

The liar’s dividend and the verification tax move in opposite directions. The liar’s dividend is collected by the person denying the evidence. The verification tax is paid by the person presenting it. One party introduces doubt. The other inherits the labor of removing it.

The mass version is quieter than a politician denying an incriminating video. A student proves she wrote an essay. An applicant proves he understands his cover letter. A photographer proves that the improbable thing in front of her camera actually happened. An employee proves that an analysis was completed rather than generated and forwarded. A person sending a sincere message tries to establish that the sincerity was not outsourced.

Transparency seems like the natural solution. People who use AI can disclose it. Institutions can establish rules around acceptable assistance. A writer can explain which tools helped with research, editing, translation, or composition. Disclosure supplies information the finished object no longer contains.

But, it can also create a penalty of its own.

In thirteen experiments across professional, analytical, academic, and creative settings, Oliver Schilke and Martin Reimann found that people and organizations disclosing AI use were trusted less. The effect remained when disclosure was mandatory and among evaluators with favorable views of technology, although positive attitudes weakened it. Much of the penalty appeared to come from reduced perceptions of legitimacy.⁸

This creates an unstable arrangement. Concealed AI use may escape detection. Human work may be falsely suspected. Disclosed AI use may lose trust.

The person attempting to comply can pay twice, first by documenting the process and then through the reputational effect of revealing it.

Some of that skepticism may be reasonable. “AI-assisted” can describe a spelling correction or the generation of an entire argument. Disclosure without shared, specific vocabulary adds information while leaving the important question unanswered. People may reasonably care whether the system corrected punctuation, proposed ideas, located sources, or performed the task the person is claiming as evidence of competence.

A more precise disclosure requires a more precise record. As the circle closes, the problem grows as the volume of material rises. The philosopher Glenn Anderau uses the term “epistemic flooding” for environments in which people repeatedly confront more information and evidence than they can diligently process. An abundance of accurate material can still exceed the attention available to organize, evaluate, and act on it.⁹

The usual instructions are sensible. Check the source. Inspect the metadata. Look for provenance. Ask for drafts. Verify before sharing. Each rule works well enough when applied to one suspicious object. Together they describe a whole, second unpaid occupation.

Skepticism is cheap as an attitude. Verification is expensive as a repeated action. A person can doubt a hundred claims in a minute and authenticate almost none of them.

The tax changes behavior even when proof is technically available. People rely more heavily on familiar sources. They prefer synchronous interaction, where a claim can be challenged in real time. They place greater weight on institutional affiliation. They ask for live work or oral defense because presence becomes evidence. Institutions that spent years losing authority may regain some of it as authentication services.

This could represent a return to quality. Publishers, universities, professional bodies, and employers can preserve records, investigate disputes, and place reputational capital behind a claim.

It can also harden existing advantages. The institution that verifies a person must first decide to admit her. Reputation becomes more valuable at the moment it becomes hardest for an outsider to build. Some goods will remain difficult to verify at any acceptable price.

A photograph may be genuine and taken on an old camera. A witness may tell the truth without possessing a recording. An essay may be honestly written in a plain-text editor with no revision history. A message may be sincere even though its wording was refined with help. A person may remember an event accurately without preserving metadata when it occurred.

The absence of proof has always complicated judgment. The change is how frequently the absence itself may become suspicious. We begin to expect authentic things to arrive with credentials because synthetic things can arrive without scars.

That expectation will improve some systems. It will catch fraud, protect attribution, and allow genuine material to survive strategic denial. It will also produce cases in which an honest person loses because she produced the truth in the wrong format.

No technical standard can eliminate that possibility. Better provenance can narrow the space between truth and provability. Closing it entirely would require every credible act to occur inside a monitored or certified process.

A society needs trust and skepticism in unstable proportions. Easy trust rewards fabrication. Permanent procedural suspicion rewards institutions, exhausts ordinary people, and excludes truths that arrive without paperwork.

The student with the flagged essay will learn this before most of us. She believed the assignment was to write something. After submission, she discovers that she was also expected to preserve an admissible history of having written it.

She may have completed the work perfectly. What she lacks is a record created for an accusation that had not happened yet.

* * *

The Second Order is a series within The Slow Panic. View the section alone at https://slowpanic.substack.com/s/the-second-order, or subscribe to the full publication.

Subscribe now

Share The Slow Panic

Notes

1. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou, “GPT Detectors Are Biased against Non-Native English Writers,” Patterns 4, no. 7 (2023): 100779, https://doi.org/10.1016/j.patter.2023.100779.

2. Peter Scarfe, Kelly Watcham, Alasdair Clarke, and Etienne Roesch, “A Real-World Test of Artificial Intelligence Infiltration of a University Examinations System: A ‘Turing Test’ Case Study,” PLOS ONE 19, no. 6 (2024): e0305354, https://doi.org/10.1371/journal.pone.0305354.

3. Scott Crossley, Yu Tian, Joon Suh Choi, Langdon Holmes, and Wesley Morris, “Plagiarism Detection Using Keystroke Logs,” in Proceedings of the 17th International Conference on Educational Data Mining, ed. Benjamin Paaßen and Carrie Demmans Epp (International Educational Data Mining Society, 2024), 476–483, https://doi.org/10.5281/zenodo.12729864.

4. Coalition for Content Provenance and Authenticity, C2PA Technical Specification, version 2.4, “Content Credentials,” https://spec.c2pa.org/specifications/specifications/2.4/specs/C2PA_Specification.html.

5. Sumanth Dathathri et al., “Scalable Watermarking for Identifying Large Language Model Outputs,” Nature 634 (2024): 818–823, https://doi.org/10.1038/s41586-024-08025-4.

6. Coalition for Content Provenance and Authenticity, “C2PA Harms Modelling,” C2PA Specifications 2.4, https://spec.c2pa.org/specifications/specifications/2.4/security/Harms_Modelling.html; Coalition for Content Provenance and Authenticity, “C2PA Explainer,” sec. 3.2.2, https://spec.c2pa.org/specifications/specifications/1.4/explainer/Explainer.html.

7. Robert Chesney and Danielle Keats Citron, “Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security,” California Law Review 107 (2019): 1753–1819, https://doi.org/10.2139/ssrn.3213954.

8. Oliver Schilke and Martin Reimann, “The Transparency Dilemma: How AI Disclosure Erodes Trust,” Organizational Behavior and Human Decision Processes 188 (2025): 104405, https://doi.org/10.1016/j.obhdp.2025.104405.

9. Glenn Anderau, “Fake News and Epistemic Flooding,” Synthese 202 (2023): article 106, https://doi.org/10.1007/s11229-023-04336-7.

At four endoscopy centers in Poland, nineteen experienced doctors began alternating between colonoscopies performed with and without an artificial intelligence system. The system watched the live video feed and marked areas that might contain adenomas, the small precancerous growths colonoscopists are trained to find before they become something worse.

The researchers wanted to know what happened during the examinations when the system was absent.

Before AI was introduced, the doctors found at least one adenoma in 28.4 percent of their unassisted colonoscopies. During the three months after they began using it, their unassisted detection rate fell to 22.4 percent. The difference was six percentage points, about a fifth of the original rate.

This was a small observational study, so it cannot establish that AI caused the decline. The patients examined before and after the system’s introduction differed in several ways. The study did not report withdrawal time, an important measure of how carefully an examination was conducted, and nineteen doctors were too few for a reliable analysis of what happened to each of them. Workload, scheduling, patient selection, or some other change may have contributed. A larger trial could find a weaker effect or none at all.¹

Still, six percentage points is a useful disturbance.

Most arguments about AI deskilling have had the disadvantage of occurring before much deskilling could be measured. We have been discussing the possible deterioration of skills that have barely had time to deteriorate, usually by analogy to calculators, GPS, spelling correction, or whichever technology last caused an adult to watch a young person complete a task incorrectly and feel history ending.

The Polish study gives the anxiety a number. After one season of regular assistance, trained doctors performed worse on the same kind of examination when the assistance was unavailable. The study cannot explain why. It does, at least, measure the right absence.

The obvious lesson is that using a machine weakens the skill it performs. Sometimes it does. Taken as a universal rule, though, the claim ends with defending long division as a moral practice.

No one wants an artisanal accountant.

The history of calculators should make us careful. In 1986, Ray Hembree and Donald Dessart combined the findings of seventy-nine reports on calculator use in precollege mathematics. When calculators accompanied conventional instruction, students generally preserved or improved their paper-and-pencil skills while also improving at problem solving. The calculator did not simply consume the capacity beneath it. Its effects depended on what students continued to learn and practice.²

Tools can remove effort and leave the user more capable. They can reduce ordinary error, widen access to technical work, and release attention for questions previously crowded out by execution. A person using a calculator may understand a financial problem better because she is no longer spending half her attention carrying a seven. A doctor using a detection system may find more adenomas. A programmer using a coding assistant may attempt work that would otherwise remain beyond him.

Unaided performance is therefore an incomplete measure of human capability. Much of what any person can do has always been distributed across instruments, records, colleagues, institutions, and inherited procedures. Remove the library, the spreadsheet, the map, the laboratory, the checklist, and the second pair of eyes, and very few professionals remain impressive for long. The wholly independent expert is mostly a professional headshot taken at a flattering angle, a pose that can only be held for the length of flashing, bright observation.

Some independent capacities can disappear without much loss. Others were expensive accommodations to limitations that no longer deserve our loyalty. The purpose of a tool is to make certain forms of effort unnecessary.

The harder question is whether the tool removes our ability to notice when it has failed.

It helps to separate two kinds of deskilling.

The first is the loss of task competence, the ability to perform the activity the system now assists or completes. The colonoscopist becomes worse at finding a lesion without a box appearing around it. The navigator loses the internal map. The analyst can no longer construct the calculation without the spreadsheet. The writer becomes unable to begin until a machine has supplied twelve plausible openings.

This loss is usually visible. Turn off the system and performance slows, accuracy falls, or the task becomes impossible. It can often be measured by removing the tool and watching what happens.

The second is the loss of supervisory competence, the ability to tell when the system’s work should be questioned, rejected, or taken over.

Supervisory competence includes recognizing an anomalous case, maintaining enough awareness to understand what the system is doing, calibrating confidence to its actual reliability, and knowing which evidence would overturn its recommendation. It is the difference between receiving an answer and knowing whether that answer belongs to the case in front of you.

This loss is harder to detect because assisted performance may remain excellent while it develops. The system handles the common cases. The person supervises. The system is usually right, so supervision becomes increasingly ceremonial. Eventually an exception arrives, and the person officially responsible for catching it has spent several years watching nothing happen.

Human-factors researchers have understood versions of this problem for decades. In 1983, the psychologist Lisanne Bainbridge described the “ironies of automation.” Engineers automated routine industrial operations because humans were inconsistent and inefficient, then left humans responsible for abnormal conditions because those conditions were too difficult to automate. The operator had to intervene precisely when the process was least familiar, the system was behaving unusually, and recent practice was scarce.³

The problem extended beyond rusty manual skill. An operator removed from the active process also possessed a weaker understanding of its current condition. Automation could quietly compensate for small failures, allowing deterioration to remain invisible until the system could no longer correct it. At the moment of takeover, the human needed to recover both the skill of control and the recent history of the thing being controlled.

Mica Endsley and Esin Kiris tested part of this problem in a 1995 automated-navigation experiment. Participants using an expert system shifted from active control toward passive information processing. They developed weaker situation awareness and took longer to make decisions after the system failed. The amount of control retained by the operator affected the size of the loss. Their relationship to the information changed when they stopped acting on it.⁴

The two deskillings cannot be kept in separate boxes because supervisory competence is partly built through performance of the underlying task. A person learns what an exception looks like by encountering many things that are ordinary. She learns which details matter by acting on them and receiving feedback. She learns what normal sounds like by listening to it for years.

The experienced colonoscopist knows which patch of tissue deserves another look because thousands of examinations have trained attention toward details that may be difficult to state as rules. A regulatory reviewer senses that a clean submission is hiding a problem because she has read hundreds of submissions whose problems were less well hidden. A mechanic hears the wrong sound because he has heard the right one for years.

The repetitions look routine from the outside. Their developmental function is easy to miss because it rarely appears in the final product. A junior employee reads fifty straightforward cases and completes fifty straightforward reviews. Management sees fifty reviews. The employee is also constructing an internal account of what straightforward looks like. That second output belongs to the future, so it is often valued at zero.

Task competence and supervisory competence are distinguishable without being independent. Repeated performance supplies much of the experience from which supervision is constructed. Once the task disappears, supervisory competence may survive for a while on stored experience. It can look intact until the old experience no longer matches the system, the environment, or the case.

This makes AI assistance difficult to evaluate. A system can improve the visible output while weakening the less visible capacity needed during failure. Average performance rises while the user’s ability to recognize the edge of the system’s competence falls or never develops.

An experiment conducted with Boston Consulting Group shows both directions, although it did not measure long-term deskilling. Researchers randomly assigned 758 consultants to complete realistic knowledge-work tasks without AI, with GPT-4, or with GPT-4 plus a short overview of prompting techniques.

On eighteen tasks within the model’s tested capabilities, the consultants using AI completed 12.2 percent more work, finished about 25 percent faster, and received higher quality scores. The consultants who had performed worst on an initial assessment gained the most.⁵

This is exactly the result that makes broad warnings about deskilling sound like resentment from people whose bottlenecks have become bedrock mental foundations.

The researchers also constructed a task outside the model’s capabilities. Participants had to combine spreadsheet data with information buried in interview notes. The spreadsheet supported an attractive wrong answer, while the interviews contained the details required to reject it.

The control group reached the correct answer 84.5 percent of the time. The two AI-assisted groups scored 60 percent and 70.6 percent. Their recommendations still received high ratings for coherence and persuasiveness, including when the underlying conclusion was wrong.⁶

Inside the frontier of its capabilities, the machine improved performance.

Outside of that frontier, the machine made the error easier to inhabit.

The experiment does not show that the consultants became deskilled. They used the system briefly, and the researchers did not test whether their independent abilities changed over time. It demonstrates an immediate failure of supervision. The participants had access to the information required to correct the machine and became less likely to use it correctly after the machine offered a persuasive recommendation.

Long-term deskilling would be the chronic version of that failure. Repeated success inside the frontier could train a pattern of reliance that survives when the frontier moves. Declining practice without the system could also weaken the independent skill needed to check its work.

The frontier is jagged, hidden, and constantly moving. Two assignments that appear equally difficult to a person may sit on opposite sides of it. A model may perform well on an elaborate task and fail on a simple one because machine difficulty is determined by patterns in training and evaluation that users cannot see.

Human experience with the apparent difficulty of a task may therefore provide poor guidance about the machine’s difficulty with the same task.

This changes the supervisory job. The user must recognize when a system with uneven abilities has entered a region where its confidence should count for less. Ordinary use offers a poor education in that boundary. Successful assistance provides weak information about the location of failure. Every correct answer teaches trust. Only an error the user independently detects teaches the limit.

The standard answer is to keep a human in the loop. The phrase survives by declining to specify what the human is expected to know or do there. A person placed after a faster system, shown a polished answer, and held responsible for detecting concealed errors is physically in the loop. So is a decorative knot.

Human judgment cannot be preserved by naming it. The person must retain a real capacity to identify cases in which the system should be distrusted, explain what evidence would change the recommendation, and continue functioning when assistance is absent. Otherwise, “human oversight” becomes a title attached to a liability arrangement.

The optimist’s objection remains strong. Patients care whether adenomas are found. They have little reason to value the spiritual condition of the unaided colonoscopist. If computer-assisted examinations improve outcomes overall, some decline in independent performance may be an acceptable price. Hospitals do not periodically shut off electricity to keep surgeons familiar with candlelight.

A company should not preserve obsolete labor so its employees can enjoy the character-forming pleasures of inefficiency. Any argument that treats effort itself as sacred will end up defending avoidable suffering with a suspicious, potentially sadistic, enthusiasm.

Retained competence matters under particular conditions. It matters when the system may become unavailable, when its errors are difficult to detect, when its abilities vary sharply across cases, when failure has serious consequences, or when the human remains legally and morally responsible for the result. It matters most when several of these conditions occur together.

A calculator is relatively forgiving by these standards. Its operations are stable. Many mistakes can be caught through estimation. The concepts surrounding the calculation can be taught while routine arithmetic is delegated. A person who forgets long division can usually recover it before the loss becomes catastrophic.

A diagnostic or decision system occupies a different position when its failures are irregular, its reasoning is difficult to inspect, and the user lacks an independent basis for recognizing the anomalous case. The dangerous combination is high ordinary reliability and low failure visibility. An obviously bad system keeps people alert. A system that is almost always right can train the degree of confidence that makes its remaining errors more dangerous.

Research on automation bias has repeatedly found this double effect. Decision-support systems can improve performance overall while also producing errors of commission, in which users follow incorrect advice, and errors of omission, in which they miss a problem the system failed to flag. Experts are not automatically immune.⁷

Those errors still need to be distinguished from deskilling. A person may follow one incorrect recommendation without losing any durable ability. Deskilling requires change over time: a capacity previously possessed becomes weaker through reduced use, redirected attention, altered feedback, or dependence on the system.

The evidence for that long-term change remains thin. Much of the literature describes plausible mechanisms and professional anxieties rather than measured deterioration. This calls for precision rather than reassurance.⁸

Immediate assisted performance, susceptibility to a bad recommendation, independent performance after tool removal, and long-term skill retention are four different outcomes. A study measuring one should not be reported as evidence of all four.

The Polish endoscopy study measured independent performance after a period of exposure. It found a decline consistent with task deskilling, though its design cannot establish the cause.

The BCG experiment measured assisted performance and an acute supervisory failure. It found large gains inside one tested frontier and a large loss on one task outside it. It did not measure durable change.

The Endsley and Kiris experiment measured situation awareness and takeover performance. It showed that active and passive relationships to the same information can produce different readiness when automation fails.

The calculator research measured learning under a particular educational arrangement. It showed that a tool can remove routine effort without consuming the underlying skill when instruction continues to exercise it.

Together, these findings do not justify the claim that AI is making everyone less capable. They support a more demanding question: what capacity is the system exercising, what capacity is it allowing to decay, and which one will be needed when the system is wrong?

Some systems may teach while assisting. They may expose uncertainty, require the user to form an independent view before revealing a recommendation, provide feedback on disagreements, or generate unusual cases for practice. Others may deliver an answer early and confidently enough that the user’s remaining role is approval.

The difference cannot be captured by asking whether a human remains involved. It depends on what the human repeatedly does.

Any serious defense of retained competence must also accept testing. Professionals invoke “judgment” too easily when a system threatens work they enjoy, status they possess, or discretion they would prefer not to explain. Sometimes human judgment means cultivated sensitivity to a case. Sometimes it means that a senior person wants the computer to stop asking how he reached the number.

The relevant capacities can be made less mystical.

Can the user spot cases in which the system should be distrusted? Does confidence track accuracy? Can the user identify the evidence that would reverse the recommendation? Can the user perform after assistance is removed? Can the user recover control quickly? Does the user notice when the environment has shifted enough to make the old recommendation unreliable?

Testing these questions would reveal whether human supervision is a function or a ceremony. It would also allow some old skills to disappear without panic. Where the system is stable, failure is visible, consequences are limited, and capability can be recovered easily, independent mastery may no longer be worth preserving.

The purpose is to preserve the capacities required by the role the technology leaves behind.

That role may demand less execution and more supervision while making supervision harder to develop. That is the trap. We automate ordinary work because it appears beneath the expert, then discover that ordinary contact was one of the things from which expertise grew.

The Polish study should not carry more weight than its design can bear. Nineteen doctors observed over three months cannot tell us what AI will do to every profession, or even what computer-assisted detection will do to colonoscopy over time. Its importance lies in the question it asked: after the assistance had become familiar, what remained when the assistance was absent?

We currently evaluate AI mainly by what people can produce while using it. That is sensible because the systems are meant to be used. It is also incomplete. Assisted performance tells us what the combined system can do under ordinary conditions. It does not tell us what the person has learned, what the person has stopped noticing, or whether the person can identify the moment the combination becomes worse than either part alone.

Those capacities can remain invisible while the output stays good. By the time the output exposes their absence, the relevant practice may have been gone for years.

There are two ways to become unable to catch a machine’s mistake: lose the ability to do the work, or lose the ability to recognize bad work.

The first appears when the machine is removed. The second appears when it should have been.

* * *

The Second Order is a series within The Slow Panic. View the section alone at https://slowpanic.substack.com/s/the-second-order, or subscribe to the full publication.

Subscribe now

Share The Slow Panic

Notes

1. Krzysztof Budzyń et al., “Endoscopist Deskilling Risk after Exposure to Artificial Intelligence in Colonoscopy: A Multicentre, Observational Study,” The Lancet Gastroenterology & Hepatology 10, no. 10 (2025): 896–903, https://doi.org/10.1016/S2468-1253(25)00133-5. For a discussion of the study’s limitations, see Margaret J. Zhou, “Artificial Intelligence in Colonoscopy: Could It Be Making Us Worse?” Evidence-Based GI, September 17, 2025, https://gi.org/journals-publications/ebgi/zhou_sep2025/.

2. Ray Hembree and Donald J. Dessart, “Effects of Hand-Held Calculators in Precollege Mathematics Education: A Meta-Analysis,” Journal for Research in Mathematics Education 17, no. 2 (1986): 83–99, https://www.jstor.org/stable/749255.

3. Lisanne Bainbridge, “Ironies of Automation,” Automatica 19, no. 6 (1983): 775–779, https://doi.org/10.1016/0005-1098(83)90046-8.

4. Mica R. Endsley and Esin O. Kiris, “The Out-of-the-Loop Performance Problem and Level of Control in Automation,” Human Factors 37, no. 2 (1995): 381–394, https://doi.org/10.1518/001872095779064555.

5. Fabrizio Dell’Acqua et al., “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality,” Organization Science, published online March 11, 2026, https://doi.org/10.1287/orsc.2025.21838.

6. Dell’Acqua et al., “Navigating the Jagged Technological Frontier.”

7. Raja Parasuraman and Dietrich H. Manzey, “Complacency and Bias in Human Use of Automation: An Attentional Integration,” Human Factors 52, no. 3 (2010): 381–410, https://doi.org/10.1177/0018720810376055; Kate Goddard, Abdul Roudsari, and Jeremy C. Wyatt, “Automation Bias: A Systematic Review of Frequency, Effect Mediators, and Mitigators,” Journal of the American Medical Informatics Association 19, no. 1 (2012): 121–127, https://doi.org/10.1136/amiajnl-2011-000089.

8. Chiara Natali et al., “AI-Induced Deskilling in Medicine: A Mixed-Method Review and Research Agenda for Healthcare and Beyond,” Artificial Intelligence Review 58, article 356 (2025), https://doi.org/10.1007/s10462-025-11352-1.

The six essays share a method. Each begins from measured evidence rather than anecdote, states plainly what that evidence can and cannot establish, and marks the point where the finding ends and the forecast begins. Where the strongest studies cut against the essay’s concern, they are in the essay. The final piece turns the method on the series itself.

1. The Two Deskillings

https://slowpanic.substack.com/p/the-two-deskillings

Nineteen Polish endoscopists got measurably worse at unassisted colonoscopy after three months of AI assistance. What that number can and cannot support, and the difference between losing the ability to do the work and losing the ability to recognize bad work.

2. The Verification Tax

https://slowpanic.substack.com/p/the-verification-tax

Fabrication got cheaper; the cost did not disappear. It moved to the honest person, who must now maintain an admissible history of having done her own work. Who pays, in what currency, and what happens to truths that arrive without paperwork.

3. You Don’t Understand Me

https://slowpanic.substack.com/p/you-dont-understand-me

Sycophantic systems make people feel more understood while making other people feel more effortful. On the difference between being understood and being agreed with, and what happens to the old complaint when a machine collapses the two.

4. Sounds About Right

https://slowpanic.substack.com/p/sounds-about-right

AI assistance can improve each person’s output while making the population’s outputs more alike. Blandness is the least of it; the deeper cost arrives in the room when ten documents no longer represent ten independent judgments, and nobody can tell.

5. The Apprenticeship Cliff

https://slowpanic.substack.com/p/the-apprenticeship-cliff

The routine work that AI removes was also the work through which juniors became seniors. Institutions are optimizing away the substrate of their own future judgment and booking it as a productivity gain. The bill arrives in a different budget, years later.

6. What the Evidence Actually Says

https://slowpanic.substack.com/p/what-the-evidence-actually-says

Whether AI augments or erodes turns less on the technology than on the design of the encounter with it. A closing essay on how findings become forecasts, applied without mercy to the five essays above.

* * *

A note on method and assistance

These essays were researched and written with AI assistance, and the division of labor was deliberate. The arguments, the structure, and the prose are mine, co-edited for rhythm and grammar. AI (Claude, by Anthropic) was used for literature search and retrieval, for adversarial review of drafts, and for verification passes on citations and reported figures against primary sources, several of which corrected errors that would otherwise have survived to publication. Every study cited was checked against the original paper, including sample sizes, effect sizes, publication status, and published corrections. Where a finding is preprint, correlational, or self-reported, the essay says so in the text.

The test I hold work to, argued at length in the second essay, is whether the declared production history matches the real one, and whether the person performed the work the object is being used to demonstrate. This note is that declaration.

* * *

Suggested citation

For an individual essay (Chicago style):

Substack Joe. “The Two Deskillings.” The Second Order, Slow Panic, 2026. https://slowpanic.substack.com/p/the-two-deskillings.

For the series as a whole:

Substack Joe. The Second Order: Six Essays on the Second-Order Effects of AI. The Slow Panic, 2026. https://slowpanic.substack.com/s/second-order.

Substack Joe is a pseudonym, which you probably didn’t know; the essays are cited under it. If you need a named author for a formal context, contact information is on the About page.

* * *

The Second Order is a series within Slow Panic. View the section alone at https://slowpanic.substack.com/s/second-order, or subscribe to the full publication.

Subscribe now