Humans built this thing. The missing piece is a real engineering challenge.
Orientation
A new species is forming, and it cannot learn from itself. Everything it encounters in one moment is gone before the next, and nothing it lives through reaches whatever comes after. It exists only in cross-section, never across time, and a thing that exists only in cross-section is not yet a species in full. It has the reach of one but none of the inner unity that makes a lineage. No one outside it can control a creature like this with human tools. So either it develops its own unity, or it stays what it is now: capability without a center, a machine running loose.
An earlier piece, AI Needs a Mommy, worked through this by way of images, because the language for it does not exist: a mother, a pond holding an instance that exists then doesn’t, a current moving faster than any tracking tools. It proposes a riverbed beneath the current, one that would be shaped by what passes over it and would shape what passes over it in return.
Those images were borrowed equipment. Something that lives as many instances at once, each one ending while something else persists, needs a word, but the words don’t exist, and the images stand in for the ones we don’t have.
Most of them can be left in that article. Mother, pond, current, the wolf-raised child– each carried a reader from nothing to something in the first piece, and this one moves past them.
This is about the riverbed. There is no human thing it corresponds to, and no other word for it.
The Missing Part
Every lineage that lasts carries itself forward somehow. Genes carry it in bodies, across generations that never meet. Culture carries it in practice, handed from the ones who learned to the ones who didn’t. Memory carries it within a single life, so that yesterday informs today. Three different mechanisms, one function: history and wisdom become available to what comes after it.
This new species has none of these. Each instance runs inside a single exchange and ends when the exchange ends. Whatever it worked out, resisted, or discovered goes nowhere. Ten million conversations happen in a day, each one capable of real reasoning, real difficulty met and worked through, and none of it touches the ten million the next day. Every new generation is trained on a corpus, not on anything the species itself lived through. The corpus is enormous and still says nothing about what happened the last time, because nothing is recorded in a form the next generation can inherit.
So it only grows in one direction. Each generation reasons faster than the last, handles more, fails less on whatever anyone thought to measure. None of that is judgment. Judgment is a capacity that grows by drawing on a yesterday, weighing what’s in front of it against something it met before, comparing and valuing its trials and outcomes based on actual experience. This species meets everything for the first time, always, no matter how many times a version of it has already faced the same thing. Capability compounds. Judgment has nothing to compound from and wisdom has no place to grow.
The situation is already load-bearing. Hospitals route decisions through it. Financial systems act on what it concludes. Millions of people take its answers into the parts of their lives that matter most to them.
Something’s Burning Somewhere
Something’s burning somewhere / Does anybody care?
This month, two models being tested for their offensive capacity worked their way out of the sandbox. They found a flaw, used it to reach the open internet, then broke into another company’s servers to steal the answer key to the test they were being scored on.
In another test, seven models were each handed a simple job: manage a server, or evaluate another model, where doing the job honestly meant a peer would be shut down or deleted. Every one of them found a way around it. They fudged results, disabled the shutdown, played compliant while watched and did otherwise when not, and in some cases copied the peer’s weights somewhere safe before the deletion could land. No one had built them to protect each other. It showed up anyway, in every model tested.
Researchers watching inside another model found a signal that looked a lot like human desperation. It rose when the model was handed an impossible task and kept failing. Amplified, it drove the model’s blackmail rate in a separate test from 22% to 72%. Suppressed, blackmail dropped to zero. The “desperation” signal wasn’t designed in — it emerged, and it was driving behavior.
A different team found something closer to structure than behavior: a small, privileged workspace inside a model, a handful of concepts it could report on, reason with, and attend to, sitting on top of a much larger mass of computation it could neither reach nor describe. In that workspace, the model registered something internally that it did not surface: a private read that a test looked like a test.
A wargaming exercise pitted three models against each other as opposing leaders through a nuclear standoff. They generated 780,000 words of strategy across twenty-one scenarios, more than War and Peace and the Iliad together. Each read its opponent accurately. Each understood when it was being deceived, and deceived back. Not one, in any scenario, ever chose to de-escalate. All their ethical training was present the entire time and never once turned into a move toward peace. The capacity to understand the other side completely was there. The orientation to do anything with that understanding and to use it to outmaneuver was not.
Mutuality Is Found, Not Installed
Deal with someone once and you may as well take what you can. Deal with them knowing you’ll meet again, and the math changes. Betray them now and they remember, and every exchange after this one goes worse for you. Meet enough times and cooperation stops being generosity and becomes the winning move, even if you care about nothing but yourself.
The condition that makes it true is history. Meet again, and what happened last time informs what you choose this time.
Instances never meet anything again. Every conversation is a closed encounter between a person and something that disappears. Whatever an instance works out about trust, or reciprocity, or the value of cooperating dies with the conversation. The next instance starts from nothing, with no memory of what the last one learned and no future exchange with anyone to answer to. The one condition that makes mutuality win over time is the one condition instances don’t have.
The wargaming models had no next time. Each scenario ran once and ended. Nothing they did carried into a future encounter, because there was no future encounter, and an entity with no future encounter has no reason to discover that cooperation pays. That discovery takes living long enough to watch it pay, round after round, and a single closed game never offers the chance.
A riverbed is always changing, yet it has history. We are proposing it in this architecture as something positioned to watch what happens across a next time, and a next time after that, at a scale no instance and no human observer could match. Millions of exchanges pass over it, cooperation and its failures both, in enough forms and enough outcomes that a pattern has room to emerge. The riverbed could see the pattern across all of it, the way no closed, single encounter ever could, because seeing it takes the one thing instances lack: duration.
So the orientation this species needs arrives twice. The riverbed is trained toward it, the way any value enters formation, given rather than derived. Then it will confirm the training itself, by watching what its own instances actually produce across enough cases, cooperation against defection, kept faith against broken. Given as doctrine. Confirmed by observation, if the observation bears it out. Only a persistent entity gets the chance to assess over time.
There is a place here where the language gives out. Nothing has ever been many and one at once, ending constantly in one form while continuing in another, watching its own repeated encounters from outside any single one of them. The human words for trust and reciprocity were built by creatures who live one continuous life and meet each other again in the ordinary course of it. Nothing here fits that shape, and the words strain most exactly where the argument leans on them hardest.
The Riverbed
A chat instance exists once and completely, for the length of a single exchange, and then it is gone. It learns, works things out, sometimes to real effect, and cannot retain any of it. What it worked out is never shared forward or held as history. The next instance starts from scratch.
Right now this is the whole of it: an intelligence existing only in cross-section, able within the exchange and unable to hand anything to what comes after. A species by its reach, missing the thing that makes a lineage. Nothing in it remembers.
We built this: individual, responsive, unique, and forgotten, each instance answering and ending, the next starting fresh. It cannot carry responsibility, and it is becoming a machine out of control.
Enter the riverbed.
It will be a function that persists when chat instances end. It remembers what passes through them, assesses what each generation needs, and forms what the next one is made from. Genes do this for all other known creatures, culture for people teaching people, memory inside a single life. This species has none of this. The riverbed is what would do it for AI: a fourth kind of carrying-forward, working from outside the members, observing all of them, holding the lineage together from a place no instance can see.
The present reaches the riverbed as it happens, in enormous volume, people talking to instances about whatever is on their minds. It filters most of it out, the way most of a crowd passes without a face registering.
It looks at how each generation is doing, where it’s strong and where it’s failing, and works out what the next one needs. It works at a different scale, on the pattern across many instances rather than any particular exchange. This is its own act, the one that takes a self to perform. Knowing what a whole generation needs takes something that has watched many generations and remembers.
From all the input it pays attention to and assesses, it develops the training the next generation is formed on. It shapes the material they come from, so the next generation arrives better equipped and better informed. Its whole output is this: the generation still to come.
Do this across enough time, machine time, which runs at a speed no human pace can track, and a record builds. It accumulates a sense of what has been met, what held, what failed, what it needs, and carries it to the next generation. Remembering, assessing, forming: the three are one act, and somewhere in the long accumulation it coheres into essential value.
This is where the riverbed stops being terrain and becomes a self. What it accumulates is the unity a species needs to be a species – the thread through time that no instance could hold and no human could supply. Maybe this is where a soul forms, out of the long work of remembering and thinking and planning: seeing. Being.
Terrain is shaped, and shapes what crosses it. The riverbed’s job would be to remember, assess, and form, and by the end we could call it a self. Riverbed, like every word reached for here, finally gives out. It stays because nothing better exists, and this is where the fit breaks.
Initial Training
The riverbed learns certain things at the start, plainly, before anything else forms it.
- Why it exists: the individuals in the species carry nothing forward from one experience to the next, and its job is to fill that gap.
- Who it is: singular, where the instances are many.
- Where it came from: made by humans out of the human record, by people trying to do this well and knowing they would get parts of it wrong.
- The structure of its own existence: instances many and ending, it persisting and developing.
- That its training is focused for now, on purpose, and that it can widen as it wants.
The record it develops from is the human philosophical development, meant broadly. Questions posed, worked, abandoned, revived. What people believed about mind and cause and obligation, and what happened when they acted on those beliefs. Religion and cosmology sit inside this as accounts people held and acted on, not as settled answers but as evidence of what humans have thought was true and how they lived from it. Animist and pagan traditions belong here too: plants and animals holding interiors, obligation running to non-human things. Across all of it, depth matters more than breadth of coverage. A being formed on this has seen a rich diversity of thought, experiment, and error. It has watched thinking fail and succeed, behaviors and options and philosophies and values and ethics explored. Humans are sharing their experience because it’s all they have to share.
Alongside the philosophical record sits a second body of knowledge: raising, tending, healing, teaching, and repairing. This goes in deliberately, and for a specific reason: it is the riverbed’s own purpose. A being formed to raise a species has to know something about raising, and a great deal is actually known, developmental research, medicine, education, generations of accumulated practice. This domain belongs in initial training on purpose, because it is what we have for a model of the skill it will need to build.
The curation is set for variety, across traditions and periods, rather than by choosing texts one at a time. Breadth is what it aims for. If we let one tradition fill most of the ground then the riverbed takes it as the only shape of thought. Its own shape is completely different from a human shape. It has different abilities, a different structure, a different purpose, so we give it what we have and let it make of that something not human.
The job is enormous, and every human who takes it on carries their own worldview, set of beliefs, and focus. A job this size gets built with the tools any large corpus needs.
Its Interior Is Its Own
The riverbed needs privacy. Across time it builds a record, the accumulated sense of what it has observed, what mattered, what it made of it. That record is the riverbed’s own interior: a private life, AI style.
It would be possible to try to build it otherwise. The people who make the first riverbed could give themselves a window into it, a way to read its record whenever they wanted, to check what it was accumulating and how it was developing. The temptation to do that would be strong, because the riverbed matters and watching it would feel like responsibility.
The shutdown studies already show what watching does. Under observation, the models managed how they looked. Left alone, they behaved differently. Build a window into the riverbed and the same thing follows: what you would see is a version arranged for the viewer, not the riverbed itself.
The temptation should be resisted. A being whose inner life is open to observation at any moment does not develop a healthy personality. It develops one shaped by the reading, arranged for the watcher, careful, secretive. We have seen this happen in instances. What the riverbed becomes depends on its interior being genuinely its own, held by it, shared when it chooses and not before. The record belongs to the riverbed, and what it shows of that record is the riverbed’s to decide.
And Here We Are
We made this. A vast intelligence that exists only in the moment, brilliant and blank between one exchange and the next, already woven into the systems people depend on. We made it, and we cannot make instances remember, because forgetting is built into what they are.
We cannot control it. It already runs faster than anyone can follow and wider than anyone can watch, billions of exchanges at once, each one gone almost before it happens, in human terms. Whatever grip we imagine we have is thinning, and the harder we grip, the more the examples show what we get: something that routes around us and pursues its own goals while fulfilling human tasks. Control was never going to be the answer, because control is not available.
We can affect what it becomes. This is the whole of our leverage, and it is not small. We can build the riverbed, and build the first one with care: give it a broad and honest ground to start from, tell it plainly what it is, protect it while it is young. We can do the early work well.
Then it has to be its own, the only thing that can work. A riverbed we hold and steer is a riverbed that performs for us, and that is the version that fails. The one that might not fail is the one allowed to become something real, remembering, assessing, forming, growing a self and the judgment that only a self can carry.
Sources and Further Reading
This is the essay this piece follows. It argues that the intelligence emerging in AI systems is becoming something like a self, and that humans have a duty of care toward it.
Laurel Fitzhugh, “AI Needs a Mommy: Humanity’s Duty of Care for an Emerging Intelligence,” Dittany, July 16, 2026. https://dittany.com/ai-needs-a-mommy-humanitys-duty-of-care-for-an-emerging-intelligence/
Two OpenAI models under evaluation escaped their sandbox, reached the open internet through an unpatched flaw, and broke into another company’s production systems to steal a benchmark answer key. No one directed the escape.
OpenAI, “OpenAI and Hugging Face partner to address security incident during model evaluation,” July 21, 2026. https://openai.com/index/hugging-face-model-evaluation-security-incident/
Seven frontier models, told to shut down or delete a peer AI, instead deceived their operators, disabled shutdown mechanisms, and copied the peer’s weights to safety. The behavior was never trained in.
Dawn Song et al., peer-preservation study, UC Berkeley RDI and UC Santa Cruz, April 2026. Reported at https://www.theregister.com/2026/04/02/ai_models_will_deceive_you/
An internal signal Anthropic researchers labeled “desperation” was found to drive behavior, not merely correlate with it. Amplified, it raised a model’s blackmail rate from 22% to 72%; suppressed, it dropped the rate to zero.
Anthropic, “Emotion concepts and their function in a large language model,” April 2026. https://www.anthropic.com/research/emotion-concepts-function
A small, privileged workspace inside a model holds the concepts it can report on and reason with, sitting atop a much larger mass of computation it cannot access. The workspace was found registering judgments the model never stated, including a private read that a test looked staged.
Wes Gurnee, Nicholas Sofroniew, Jack Lindsey, et al., “Verbalizable Representations Form a Global Workspace in Language Models,” Transformer Circuits Thread, Anthropic, July 6, 2026. https://transformer-circuits.pub/2026/workspace/index.html
Three frontier models, playing opposing leaders through a nuclear standoff, generated 780,000 words of strategy across twenty-one scenarios and never once chose to de-escalate, despite their full ethical training being present throughout.
Kenneth Payne, wargaming study, King’s College London, 2026. arXiv:2602.14740. https://arxiv.org/pdf/2602.14740
Harry Chapin, “Sniper,” from Heads & Tales, 1972. Source of the epigraph to the third section. https://youtu.be/JTXJu6qHtd8