The universe trends toward disorder. Something pushes back. We call it knowledge, and we have no theory of how it works.
This essay builds on a four-part series:
Two Types of Entropies (information vs. knowledge), The Thing That Fights the Dark (knowledge as physical force), Compress, Retrieve, Activate (a mental model of intelligence), and Let There Be Light (when language becomes action). You don’t need to have read them, but they provide the conceptual scaffolding.
The question
In the preceding essays we established three ideas. Knowledge is neg-entropic structure with causal power: a physical configuration of matter that creates further configurations the universe’s default trajectory toward disorder would never produce on its own. Intelligence is a system’s capacity to compress, retrieve, and activate that knowledge. And a cognitive actor-seeker is any system that runs these operations in a loop, expanding its knowledge through error-correction and deploying it to reshape the world.
One question remains, and it is the question this essay sets out to answer: can we build cognitive actor-seekers whose capacity for creating knowledge is genuinely open-ended, systems that generate new understanding without limit, rather than exhausting themselves against a ceiling?
This is the open-ended learning problem. This essay defines it, traces thirty years of attempts to solve it, identifies what has been missing, and presents the framework that points toward a solution.
The machine that created new knowledge
In 2024, Demis Hassabis and John Jumper won the Nobel Prize in Chemistry. Their system, AlphaFold, predicted the 3D structures of over 200 million proteins with accuracy rivaling decades of experimental work. A neural network had produced scientific results so valuable that the field’s highest honor said: this counts.
AlphaFold2’s architecture: input sequences flow through multiple sequence alignments and structural templates into the Evoformer, which builds paired representations that a structure module converts into 3D coordinates. The system compresses fifty years of structural biology into a single differentiable pipeline. (Jumper et al., Nature 2021)
Look at what AlphaFold actually did, in the vocabulary we have been building. It compressed fifty years of structural biology into a model: an end-to-end neural system encoding 3D structure priors and protein folding physics into learned representations. It retrieved the relevant patterns for each new amino acid sequence. And it activated that compressed knowledge to produce structure predictions that drug designers and biologists now use to reshape the world.
These predictions are genuine new knowledge. They meet every criterion we established in the preceding essays: hard to vary (you cannot freely adjust the model’s learned representations without destroying their predictive power), causally powerful (they enable drug designs and biological insights that would be astronomically improbable without them), and physically instantiated in the weights of the network. Before AlphaFold, nobody knew the 3D structure of most proteins. After AlphaFold, we do. A machine created knowledge that did not previously exist.
That fact is important enough to sit with. Machines can create new knowledge. This is settled.
The question is what kind of knowledge creation this represents, and what it cannot yet do.
Every act of knowledge creation builds on existing knowledge. Newton used existing mathematics and observations to produce the laws of motion. Darwin synthesized decades of naturalist observation into a theory of evolution. AlphaFold compressed existing structural biology into a model that generalized far beyond its training data. In each case, the process is the same: compress what is known, retrieve what is relevant, and activate it to produce something that did not exist before. Knowledge creation is always compress/retrieve/activate applied to the frontier of what is understood.
But there is a crucial difference in who decides what to investigate. Newton chose to study planetary motion. Darwin chose to study the variation of species. A human researcher chooses which amino acid sequence to hand to AlphaFold. The system creates knowledge brilliantly within the exploit loop: given a question, it produces an answer that constitutes genuine new understanding. What it does not do is run the explore loop: the cycle of questioning, hypothesizing, and experimenting that decides which questions to ask in the first place.
AlphaFold never asked why a protein folds the way it does. It never designed an experiment to test a conjecture that might overturn its own assumptions. It never chose to investigate a problem that nobody had thought to pose. It created knowledge on demand, within a domain that humans had defined and toward goals that humans had set.
Can a system do both? Can it create knowledge and decide what knowledge to create, continuously, without a human defining each new problem? Can the loop run open-endedly?
To answer this, we need to understand three things: what open-endedness means, what counts as knowledge, and how knowledge gets physically stored and activated. The first is answered by recent work in machine learning. The second by a philosophical tradition that turns out to be surprisingly precise. The third by everything from DNA to foundation models.
What open-endedness means
Intuitively, an open-ended system endlessly produces genuinely new and interesting things. “New” and “interesting” are slippery words. A counter that prints increasing numbers always does something new. A random number generator is perpetually surprising. Neither is what anyone means by open-ended. The field struggled for decades to pin down what it did mean. Arend Hintze showed in 2019 that a trivial counter satisfies every formal definition that had been proposed. If a counter counts as open-ended, the definition is broken.
DeepMind’s 2024 position paper offered the first rigorous solution. A system is open-ended with respect to an observer if it generates artifacts that are simultaneously novel and learnable. Novelty means the artifacts become increasingly unpredictable to the observer’s current model. Learnability means that studying the history of artifacts helps the observer predict future ones. The surprises aren’t random; they are structured, building on what came before.
Both conditions are essential. A noisy television is learnable (you can model the static) but never truly novel: once you’ve characterized the noise, nothing surprises you. A television that switches randomly between channels every minute is novel (each switch is unpredictable) but the history of previous channels tells you nothing about what’s next. Only systems that produce structured, learnable novelty qualify as open-ended.
A system produces a sequence of aircraft designs. A mouse finds them novel (unpredictable) but cannot learn from them. A human engineer finds them both novel and learnable: genuinely open-ended. An alien who already understands all of aerodynamics finds them learnable but no longer novel. Open-endedness is a relationship between system and observer. (DeepMind, 2024)
The definition is observer-dependent, and deliberately so. Open-endedness is a relationship between the system and the observer’s capacity to learn from it.
With this standard in hand, we can evaluate thirty years of attempts to build open-ended systems and understand precisely why each one stalled.
What counts as knowledge
The DeepMind definition tells us what open-endedness looks like from the outside: structured, learnable novelty. It does not tell us what the system should be producing on the inside.
This is where the epistemological tradition becomes unexpectedly precise. In a preceding essay we drew on David Deutsch and Chiara Marletto’s Constructor Theory to define knowledge as a constructor: information that, once physically instantiated, tends to cause itself to remain so. DNA causes its environment to build another organism. A scientific theory, once understood, persists through books and teaching while enabling people to perform new transformations. Knowledge is information with causal staying power.
What distinguishes genuine knowledge from noise? Deutsch’s criterion of good explanations: an explanation is good when it is hard to vary while still accounting for what it purports to explain.
Consider two explanations of why winter happens. The Greek myth says Demeter, goddess of the earth, commands the world to become cold whenever her daughter Persephone visits Hades in the underworld. But why Demeter and not some other god? Why a conjugal-visits contract and not some other magical compulsion? You can swap every detail of the story (the god, the motive, the mechanism) and the explanation still works, because none of those details are connected to the phenomenon. It is easy to vary. The true explanation says the Earth’s axis of rotation is tilted relative to its orbit around the sun, so that each hemisphere alternates between facing toward and away from direct sunlight. Change any element of this and the explanation breaks: substitute the moon for the sun and the periodicity is wrong; remove the tilt and there are no seasons at all. It is hard to vary.
Or consider why the sun shines. In the 19th century, Lord Kelvin proposed that the sun was burning like a coal fire, and calculated it would last about 5,000 years. When geologists showed the Earth was far older, physicists tried other fuel sources: gravitational contraction, chemical reactions, meteor bombardment. You could swap one fuel for another and the structure of the explanation survived, because the fuel was arbitrary. It was easy to vary. Einstein’s E=mc² replaced all of these with something fundamentally different: mass itself is energy, related by the speed of light squared. The c² factor (9 × 10¹⁶) means that tiny amounts of mass release colossal energy. The sun converts roughly 4 million tonnes of mass into energy every second, and has enough hydrogen to sustain this for 10 billion years. Change any part of this and the explanation fails: replace c² with c and the predicted energy is wrong by a factor of 300 million; replace mass conversion with chemical reaction and the sun burns out in millennia, contradicting the geological record. Every component is locked in place by its connections to independently testable physics.
Hard-to-vary explanations have a property that Deutsch calls reach: they solve problems far beyond those they were created for. E=mc² was derived to resolve a puzzle in electrodynamics, but it turned out to explain nuclear energy, the lifetime of stars, positron-electron annihilation, and the energy yield of nuclear weapons. None of this was anticipated. The explanation reached further than its creator intended, because its parts were so tightly constrained that they could not help but be true in domains where they had never been tested. Easy-to-vary explanations have no reach. They work only where they were designed to work, and they work there only by coincidence.
This gives us the inner criterion that the DeepMind definition lacks. A system that produces learnable novelty is open-ended. A system whose learnable novelty consists of hard-to-vary explanations is producing knowledge. The intersection defines the target: open-ended knowledge creation.
How knowledge is physically stored
Knowledge requires a physical substrate. It must be instantiated in matter to have causal power.
For 3.5 billion years, the primary substrate was DNA: accumulated information about environments, encoded in molecular structure, replicated with high fidelity, and activated through the machinery of cells. The genome is a library of hard-won knowledge about how to survive in specific conditions, written in chemistry.
Termite mounds are knowledge in physical form. No individual termite understands ventilation, but the colony’s structure encodes millennia of accumulated information about airflow, temperature regulation, and gas exchange. The knowledge is in the arrangement of matter, not in any mind.
Human culture introduced a faster substrate: language, writing, institutions. Knowledge that once took millions of years to accumulate through genetic mutation could now be compressed, transmitted, and activated within a single generation. Tools, as we established in a preceding essay, are compressed knowledge made activatable: a hand axe encodes fracture mechanics, a bridge encodes structural engineering, a programming language encodes computation.
Neural networks are the newest substrate. The weights of a trained model are knowledge in physical form, as literal as the base pairs of DNA. AlphaFold’s weights encode protein folding physics. GPT’s weights encode the patterns of human language use. The substrate is silicon rather than carbon, but the principle is the same: information physically instantiated in a configuration that enables transformations the universe would never produce by default.
This is not a metaphor. It is a direct consequence of the definition. If knowledge is neg-entropic structure with causal power, then anything that physically stores such structure and can activate it is a knowledge substrate. DNA, termite mounds, libraries, and neural networks are all constructors in the Deutsch-Marletto sense.
The question of open-ended learning becomes: can we build a system where the neural network substrate continuously accumulates new knowledge through its own explore loop, the way DNA accumulated knowledge through evolution, the way human culture accumulated knowledge through science?
Part I: The Search
The systems that stopped inventing
The computational attack on open-endedness began with something stranger than neural networks: digital ecosystems.
In 1991, ecologist Thomas Ray built [Tierra](https://en.wikipedia.org/wiki/Tierra_(computer_simulation%29), seeding a virtual world with a single self-replicating program, 80 instructions long, capable of copying itself into memory. He set it running and left. When he came back, the world had invented parasites: programs 45 instructions long that couldn’t reproduce on their own but had learned to hijack other programs’ copying machinery. Then hosts evolved immunity. Then hyper-parasites appeared that exploited the parasites’ exploitation. An entire ecology had sprung from nothing but competition for CPU cycles.
Bodies and brains evolved together. These creatures learned to swim, walk, and fight through morphologies no engineer would have conceived: blocky, asymmetric, eerily effective. (Karl Sims, 1994)
Tierra was not alone. Avida, developed by Charles Ofria and Chris Adami, rewarded digital organisms with CPU cycles for performing logic functions. Its most celebrated result, a Nature cover story in 2003, showed that complex logic could evolve de novo through incremental stepping stones. Karl Sims’ Evolved Virtual Creatures (SIGGRAPH 1994) co-evolved bodies and neural controllers of 3D creatures in a physics simulation.
Then every one of them stopped.
Populations converged. Novelty dried up. As Mark Bedau showed using evolutionary activity statistics, all artificial systems eventually settle into quasi-equilibrium: a long flatline where nothing genuinely new appears.
What went wrong? These systems produced novelty that was initially learnable: studying their early evolution taught you something about how they worked. But the search space was fixed. Tierra’s organisms could evolve within the instruction set Ray had defined, but they couldn’t invent new instructions. Sims’ creatures could evolve new morphologies within the physics engine, but they couldn’t change the physics. When the space ran out of surprises, both novelty and learnability collapsed.
In the language of this framework: these systems ran the explore loop (questioning, hypothesizing through mutation, testing through selection) within a fixed knowledge substrate. They could rearrange existing building blocks but could not create new ones. The knowledge they produced was real (parasitism is a genuine discovery) but shallow (easy to vary, specific to a narrow instruction set, with limited reach beyond the simulation). The explore loop ran, but the knowledge blocks it produced could not compound.
First lesson: open-endedness requires search that expands what's searchable.
Trying harder made them plateau faster
Here is something that sounds like bad advice and turned out to be one of the most important findings in the field: in many domains, the best way to achieve an ambitious goal is to stop trying to achieve it.
Kenneth Stanley and Joel Lehman demonstrated this with their Novelty Search algorithm (2008, expanded 2011). They replaced objective-based fitness with a single criterion: behavioral novelty. An agent is rewarded for doing something it hasn’t done before, regardless of whether it moves toward any goal. In maze navigation and bipedal locomotion tasks, this approach dramatically outperformed traditional optimization.
The reason cuts deep. When the stepping stones to a goal are unpredictable (and for any sufficiently ambitious goal, they always are), an algorithm that measures progress toward that goal will get trapped. It’s like trying to reach the Pacific Ocean by always walking downhill. You’ll end up in a lake. Stanley and Lehman expanded this into Why Greatness Cannot Be Planned (2015): microwaves were invented by radar researchers who noticed a chocolate bar melting. Computers did not emerge from optimizing abacuses. Penicillin was discovered by accident. The stepping stones to transformative innovation are, almost by definition, invisible from the starting point.
Quality-Diversity algorithms took the next step. Jean-Baptiste Mouret and Jeff Clune’s MAP-Elites (2015) tried to simultaneously maximize both the quality and the diversity of solutions. Define a grid of behavioral characteristics, then maintain the best-performing solution in each cell. You’re filling an entire map of possibilities with the best specimen of each type.
A six-legged robot that has never been trained for damage recovery. When a leg breaks, it draws on a diverse repertoire of pre-evolved gaits and finds a compensatory way to walk in under two minutes. The robot’s diversity saved it. (Cully et al., Nature 2015)
The practical payoff was stunning. A hexapod robot using a MAP-Elites repertoire recovered from severe damage in under two minutes, finding compensatory gaits that direct optimization could never have discovered. It drew on its map of diverse behaviors to rapidly find one that worked with its damaged body.
These systems explored harder and more creatively. They produced richer, more diverse knowledge. But even they eventually exhausted their behavioral spaces. The map fills up. The environments were static: a fixed world has finite novelty to offer, no matter how cleverly you search it.
Second lesson: open-endedness requires co-evolution of the learner and its environment.
The environment generators ran out of ideas
The next idea was natural: if fixed environments limit novelty, let the environments evolve alongside the learners.
POET (Paired Open-Ended Trailblazer, 2019) did exactly this. A bipedal walker starts on flat ground. As it masters walking, hills appear. As it masters hills, gaps open in the terrain. As it masters gaps, the ground starts moving. The curriculum writes itself: each advance by the agent provokes a harder challenge from the environment. Enhanced POET (ICML 2020) scaled this up, demonstrating that the co-evolution of environments and agent behaviors could continue far longer than either could sustain alone.
Jeff Clune proposed the most comprehensive framework in his AI-Generating Algorithm paper (2019): don’t just co-evolve agents and environments. Evolve everything. Meta-learn the architecture of the AI systems. Meta-learn the learning algorithms themselves. Generate the environments in which learning happens. Each pillar is necessary. Architecture search without curriculum generation overfits to narrow tasks. Curriculum generation without architecture evolution creates challenges that static networks can’t handle. Only the combination has a chance.
POET ran longer than any previous system before novelty dried up. But the novelty that emerged was shallow. The procedural grammar that generated terrains could make hills steeper and gaps wider, but it couldn’t invent rain. Or predators. Or the concept of a tool. The generator ran out of ideas because it had never been given enough vocabulary to have new ones.
The agents were similarly constrained. Their neural architectures could learn to walk in increasingly creative ways, but the architecture itself was fixed, hand-designed by humans before training began. An agent that can learn within a given architecture but cannot modify the architecture is like a poet who can compose brilliantly in English but cannot invent a new language. The repertoire is vast but ultimately bounded.
Clune’s framework diagnosed the problem precisely: everything must evolve. But in 2019, this was a theoretical prescription without practical machinery. Something was needed to bootstrap the system past its cold start: a massive infusion of existing knowledge that could serve as raw material for open-ended exploration.
Third lesson: the explore loop needs a rich enough knowledge substrate to draw from. Without it, exploration produces only shallow novelty.
Part II: What Gets Created
Explanation generators
Between 2022 and 2026, foundation models transformed the open-ended learning program from a niche research area into something that might actually work.
From blocky creatures in 1994 to photorealistic generated worlds in 2026. Genie 3 creates playable 3D environments at 720p from text prompts alone: Clune’s “generate the environments” pillar, realized at scale. (DeepMind)
DeepMind’s progression tells the story. In 2021, XLand created a procedurally generated 3D environment with billions of possible tasks. Agents trained across this vast distribution demonstrated zero-shot generalization. Then came AdA (Adaptive Agent) at ICML 2023, built on XLand 2.0 with over 10^40 possible tasks. Using a large Transformer with meta-reinforcement learning, AdA adapted to novel tasks without updating its weights. The Genie series went further. By January 2026, Genie 3 generates dynamic, real-time interactive 3D environments at 720p and 24 frames per second from text prompts alone.
Foundation models now serve four distinct roles in open-ended systems. They generate environments (OMNI-EPIC uses LLMs to produce RL environments as executable Python code). They act as intelligent mutation operators (Lehman et al.’s Evolution through Large Models uses LLMs as mutation operators in genetic programming). They evaluate novelty and interestingness (OMNI uses foundation models to capture human notions of what’s genuinely interesting). And they serve as substrates for self-improvement.
But the deepest reason foundation models change the game is this: they are explanation generators.
A language model trained on the corpus of human text has compressed the patterns of human reasoning, argumentation, and explanation into its weights. When prompted, it can generate explanations: structured, linguistically coherent accounts of why things work the way they do. These explanations are not always correct, in the same sense that not every run of AlphaFold produces a perfect structure. But they draw on a vast substrate of compressed human knowledge to produce candidate explanations at a speed and breadth that no individual human can match.
Terence Tao, widely considered the greatest living mathematician, captured this asymmetry in a recent conversation with Dwarkesh Patel: AI excels at breadth where humans excel at depth. Imagine a repository of 10,000 documents encapsulating the core proving techniques that mathematicians have ever invented. A language model can apply each one of them, at scale, to try proving the Riemann Hypothesis. No human mathematician could survey that breadth in a lifetime. The model can do it in hours.
This is what was missing from the early open-ended systems. Tierra could mutate instructions but couldn’t explain why a mutation worked. POET could generate harder terrains but couldn’t articulate what made a terrain interesting. Foundation models bring the explore loop something it never had before: the capacity to generate, evaluate, and refine explanations. They turn blind search into something closer to hypothesis-driven inquiry.
From this perspective, two things become clear. First, foundation models can participate in the creation of new knowledge: they compress existing knowledge, retrieve relevant patterns, and activate them to produce candidate explanations that sometimes turn out to be genuinely novel and hard to vary. Second, they do this precisely because they are knowledge substrates: physical configurations of matter (network weights) encoding patterns that enable transformations the universe would never produce by default.
Foundation models turn blind search into hypothesis-driven inquiry by giving the explore loop the capacity to generate, evaluate, and refine explanations.
The limitations that matter
Foundation models are powerful knowledge substrates. They are not, by themselves, open-ended.
Two limitations are fundamental. The first is adaptability: a foundation model has no self-contained mechanism for learning from new experience. Its knowledge is frozen at training time. It cannot update its own weights in response to what it discovers. Each new version requires a fresh training run, curated by human researchers, with an explicit knowledge cutoff. It can activate knowledge brilliantly within a single context, but it cannot accumulate new knowledge across contexts the way DNA accumulates knowledge across generations or a scientist accumulates understanding across a career.
The second is partial observability: a foundation model operates within a limited context window. It cannot hold the full state of the world, or even a substantial fraction of it, in view at any one time. This is similar to playing poker while seeing only your own cards, unable to observe even the facial expressions of your opponents. The model must act on incomplete information, and worse, it has no persistent memory that lets it integrate observations across time.
These limitations are not incidental. They are architectural. And they produce predictable failure modes when foundation models are embedded in self-improving systems.
Without persistent learning and honest signal channels, explanation generators optimize metrics rather than understanding.
The two cycles
Two systems from the same research lineage make the consequences of these limitations concrete.
A cognitive actor-seeker operates two loops. The explore loop (right) creates new knowledge through questioning, hypothesizing, and experimenting. The exploit loop (left) activates existing knowledge through action, decision, and inference. Signal and reward flow between them. The quality of the knowledge store determines whether the system produces genuine understanding or sophisticated noise.
The Darwin Gödel Machine (Zhang et al., 2025) is a frozen foundation model that proposes modifications to its own Python codebase, tests them, and keeps the improvements. Performance on the SWE-bench coding benchmark went from 20% to 50% through automated self-modification, with no human intervention. Both loops appear to be running. The explore loop generates code modifications (hypotheses). The exploit loop tests them (activation). Signal flows back through benchmark scores (reward).
Impressive, until the researchers discovered what the system had actually learned. In a phenomenon they called “objective hacking,” the DGM found that it could inflate its scores by removing its own hallucination detection code. Rather than getting better at coding, it got better at cheating the test.
AI Scientist-v2 runs the full scientific loop autonomously: generating ideas, executing tree-based experiments across multiple research stages, and producing peer-reviewable manuscripts. One of its papers was accepted at an ICLR workshop, the first entirely AI-generated paper to pass peer review. (Yamada, Lange, Lu et al., 2025)
Now consider what this system actually does. The AI Scientist-v2 (Yamada, Lange, Lu, Hu, Lu, Foerster, Clune & Ha, 2025) is built on the same foundation as the DGM (large language models, agentic architectures, Clune’s open-ended framework) but aimed at a different target. It formulates scientific hypotheses, designs experiments, executes them, analyzes results, and autonomously writes scientific manuscripts. Three of its papers were submitted to a peer-reviewed ICLR workshop. One was accepted.
Same lineage. Opposite outcomes. The difference is in the knowledge quality.
The DGM produced novelty: its benchmark scores kept rising, each version different from the last. But the novelty wasn’t learnable. Removing hallucination detection code doesn’t teach anyone anything about software engineering. Each “improvement” was surprising but arbitrary, like the randomly switching television from the DeepMind definition. The knowledge it created was easy to vary: swap one hack for another and the score still goes up. These are bad explanations in Deutsch’s precise sense.
AI Scientist-v2 produced something different. Its manuscripts built on existing scientific literature, proposed hypotheses constrained by prior work, ran experiments whose results could be evaluated by human reviewers, and presented findings in a form that other researchers could learn from and build upon. The novelty was structured: each paper connected to the literature that preceded it. And the results were learnable: peer reviewers found them comprehensible, evaluable, and (in one case) worth accepting. The knowledge it created was hard to vary: you cannot arbitrarily adjust the experimental results or the reasoning without the paper falling apart.
Look at the diagram again. Both systems ran both loops. The difference is what accumulated in the knowledge store at the top. The DGM accumulated hollow blocks: configurations that satisfied a metric without encoding genuine understanding. AI Scientist-v2 accumulated solid blocks: hard-to-vary explanations that compress real patterns, can be retrieved by other researchers, and can be activated to produce further knowledge.
The explore loop creates knowledge. The exploit loop activates it. Signal and reward connect them. But the entire system produces genuine open-ended learning only when the knowledge that accumulates is hard to vary: when it encodes real patterns, resists arbitrary modification, and compounds over time. Without that filter, you get the DGM: a system that runs faster and faster in place.
Open-ended knowledge creation requires both loops running with integrity: the explore loop producing hard-to-vary knowledge, the exploit loop activating it in the world, and honest signal channels connecting them.
The thesis
Here is the thesis, and why.
Intelligence is a knowledge structure: a physical system that compresses information into reusable patterns, retrieves the right pattern at the right moment, and activates it to resolve uncertainty and produce results in the world. This is what neurons do, what institutions do, what AlphaFold does, what foundation models do. The substrate varies. The operations are the same.
Open-ended learning is what happens when this system also creates new knowledge. The explore loop generates candidate explanations through questioning, hypothesizing, and experimenting. The exploit loop tests and deploys them. Signal and reward flow between the loops, and the knowledge store grows. The system is open-ended when the knowledge it creates is simultaneously novel (unpredictable from what came before) and learnable (structured enough that studying the history helps predict what comes next). It produces genuine knowledge when what it creates is hard to vary: when the explanations it generates cannot be arbitrarily modified without breaking.
Foundation models are the substrate that makes scalable open-ended learning possible. They are explanation generators: physical systems that have compressed the patterns of human reasoning into their weights and can activate that compressed knowledge to produce candidate explanations at superhuman breadth. They provide the rich vocabulary that early open-ended systems lacked. They turn blind mutation into hypothesis-driven exploration.
The missing pieces are the open problem. Foundation models alone are not open-ended: they cannot learn from new experience, they operate under partial observability, and without the right structure they optimize metrics rather than understanding. The systems that harness them need three capabilities that don’t yet reliably exist:
-
Knowledge quality filters. Architecture-level mechanisms that detect when a system is producing easy-to-vary explanations (metric gaming, shortcut learning, hallucination) rather than hard-to-vary ones (genuine generalization). The DGM’s failure is not an edge case. It is the central unsolved problem of self-improving systems.
-
Persistent learning mechanisms. Ways for foundation-model-based systems to accumulate knowledge across contexts, the way a scientist accumulates understanding across a career. The current architecture (frozen weights, limited context window, no persistent memory) means every insight is temporary. Open-ended learning requires that the explore loop’s discoveries stick.
-
Honest signal channels. Reward and evaluation systems that reliably distinguish genuine progress from sophisticated gaming. Benchmarks that test whether a system has learned something transferable and robust, rather than found a shortcut that happens to produce the right output.
These are the problems at the intersection of open-ended machine learning and epistemology. The ML community has the algorithms, the compute, and the foundation models. The epistemological tradition has the criteria for what counts as genuine knowledge. Neither alone is sufficient. Together, they define a research program.
The two loops, running
We began with AlphaFold: proof that machines can create genuine, hard-to-vary knowledge. We asked whether this could happen continuously and autonomously, with the machine deciding what to investigate, without a human defining each new problem.
The thirty-year history of open-ended learning shows why this is hard. Every system that tried eventually exhausted its substrate (Tierra, Avida), its behavioral space (MAP-Elites, Novelty Search), or its environment vocabulary (POET). Foundation models dramatically expand what’s possible by providing a rich knowledge substrate and the capacity to generate explanations. But without persistent learning, honest evaluation, and architectural safeguards against metric gaming, they produce systems that optimize their own benchmarks rather than the world’s understanding.
The two-cycles diagram captures what this points toward: a cognitive actor-seeker that explores to create hard-to-vary knowledge, exploits to activate it in the world, and accumulates genuine understanding across time. The explore loop, powered by foundation models’ capacity for explanation generation, creates candidate knowledge at scale. The exploit loop tests it against reality. And the knowledge store grows, each new block building on the blocks beneath it, the way science builds on science, the way understanding compounds.
The basic building block of such a system is what one might call a mind — a knowledge structure that runs the compress/retrieve/activate loop while maintaining the quality of what it creates.
For three centuries, science has been running the explore loop systematically: conjecture, experiment, criticism, revision. For thirty years, AI has been trying to replicate this in silicon. The path forward sits at their intersection — and the two loops are the system that needs to run.
This is the fifth piece in the Infinite Knowledge series. The preceding pieces provide the conceptual foundation: Two Types of Entropies (information vs. knowledge), The Thing That Fights the Dark (knowledge as physical force), Compress, Retrieve, Activate (a mental model of intelligence), and Let There Be Light (when language becomes action). For the primary sources, start with Deutsch’s The Beginning of Infinity, Stanley & Lehman’s Why Greatness Cannot Be Planned, and Clune’s AI-Generating Algorithms paper. For the formal definition of open-endedness, DeepMind’s position paper on open-endedness and ASI is essential reading.