Somewhere in the Amazon, for at least the past few thousand years, indigenous communities have been processing manioc the same careful way. Peel, grind, rinse, and then a multi-day waiting period that looks, to any modern observer, like pointless delay.
The waiting is not pointless. Untreated manioc contains cyanogenic compounds. The processing destroys them, and the compounds destroy a human body slowly. The harm surfaces decades later, as a crushing neurological condition called konzo, in the children and grandchildren of families who eat untreated manioc. No individual in the tradition could have discovered that causal chain by watching, because the effect is separated from the cause by thirty years and two generations.
The communities that keep the ritual do not know any of this. When anthropologists asked, the explanations on offer were stories about spirits or tradition, and the practice itself persists because it was inherited, not because it was understood. The anthropologist Joseph Henrich, who built his career on cases like this one, drew the conclusion the case forces. The ritual is knowledge that no member of the culture holds. It is a constraint, fitted across generations of trial and error, that keeps a whole people from slowly poisoning their children without anyone needing to understand why.
A civilisation is a belief system. Its parameters are institutions, laws, religions, scientific paradigms, and the tacit knowledge embedded in customs whose original rationale has been forgotten. Its training data is the historical record, oral tradition, and the lived experience of its members. Its predictions are the bets it makes about how to organise society and respond to crises. A civilisation does not “have” beliefs the way a person does. It does have a model, distributed across millions of people and materialised in built environment and codified text, and that model produces interpretations of the present and predictions about the future as surely as your gut produces a feeling about a stranger across the room. The civilisation knows things its members do not, because it has trained on data no single person has access to.
The companion piece made the case that your beliefs are the output of a model you cannot directly see, fitted to the slice of experience you happen to have lived, and that every such model faces the same trade-off. Tighten it and you gain resolution inside your lane while losing robustness when the world moves. Loosen it and you gain robustness while losing resolution. No model gets both. This piece takes that constraint out of your head and shows it operating at every scale.
The failure mode is familiar. Kodak’s belief model said the future of imaging was chemical. Nokia’s said the future of phones was hardware excellence. Blockbuster’s said the future of video rental was retail. Each was fitted, exquisitely well, to a regime that was real for a long time and then stopped being real on a timeline the model could not see. From the outside the collapses looked like strategic failures. From the inside, they were belief-model failures. The organisation’s collective lens delivered confident, crisp predictions about a world that had already moved on.
This is what makes overfitting the pathology of success. The more competent a system becomes within its training distribution, the more brittle it becomes outside that distribution. It holds for surgeons, for NASA (which progressively normalised O-ring anomalies until the Challenger disaster), and for entire civilisations that build increasingly elaborate institutions around assumptions that eventually stop being true. The mirror failure is underfitting, the pathology of oversimplification. A belief model too coarse for its environment is perpetually surprised, whether the model belongs to a new employee, an under-resourced regulator, or a central planner trying to govern a complex economy with a handful of targets.
The mechanism has a name. James March formalised it in a 1991 paper. Organisations face a tension between exploitation (getting better at what they already do) and exploration (searching for what they might do instead). Exploitation is the low-variance path, reliable and efficient and measurable. Exploration is the high-variance path, uncertain and expensive and often fruitless. March’s central finding was that organisations systematically over-invest in exploitation because its returns are faster and more legible. They get spectacularly good at a version of reality that is slowly becoming obsolete.
But the manioc case is the stranger and more instructive one, because the belief model at work there is not failing. It is working so well that nobody can see it working. Strip the constraints away, through colonisation, industrialisation, or the confident replacement of a tradition with a more “rational” practice, and the belief model loses regularization it could not articulate. The system overfits to a new regime in ways that take decades to surface as harm. The rest of this essay is about what these scaled-up belief models do, where they live, how fast they can change, and what it takes to keep them honest.
Where belief models live, and how fast they update
Belief models do not just sit at different scales. They live in different substrates and update at different speeds, and that difference shapes the kinds of error each is prone to.
An individual stores belief models in neural patterns, schemas, and embodied skills. The model is fast to access, richly contextual, and impermanent. It dies with the person, or degrades with age and trauma. It can also change in an afternoon.
An organisation stores belief models in processes, SOPs, databases, and the informal norms that make “how things actually get done” legible to insiders. The model is slower to update than any individual’s, more robust to individual departure, and far more prone to ossification. A process written for a good reason in 2005 still runs in 2026 because no one remembers why it exists and no one wants to be the person who kills it. An organisation can pivot, painfully, in a quarter.
A civilisation stores belief models in laws, constitutions, religions, languages, scientific literatures, and the vast repository of tacit practices transmitted by apprenticeship. The model is the slowest to update, the most robust to individual disruption, and the most susceptible to catastrophic forgetting. The Late Antique loss of Latin literacy and concrete engineering, the Ming destruction of Zheng He’s navigation records, and the Cultural Revolution’s systematic erasure of teachers and libraries are all examples of the same failure mode. A civilisation updates its deep parameters over decades or centuries, if at all.
The bias-variance trade-off plays out differently at each timescale. Individual models have high variance and relatively low bias. The lens can be re-aimed quickly. Civilisational models are the opposite. Their assumptions are stable, often centuries old, and resist revision precisely because they hold so much else up.
The asymmetry produces a characteristic failure mode. Individual-level variance gets damped by organisational and civilisational bias. A junior analyst at a bank notices a risk the models miss. Her concern gets absorbed by routines fitted to the last regime. Those routines sit inside regulatory frameworks fitted to the crisis before that. Three belief models, each overfitting on its own clock, each muffling the signal that might correct it. The system as a whole is brittle in a way no individual model is.
Exploration looks different at each scale too. An individual explores by trying a new skill, reading outside her field, or changing jobs. An organisation explores by funding R&D, running skunkworks, or hiring from outside the industry. A civilisation explores by maintaining religious pluralism, free inquiry, dissent, competitive jurisdictions, and a deliberate diversity of approaches to the same problem. The key insight from Elinor Ostrom’s work on commons governance and the broader polycentric-governance literature is that ensembles of diverse, semi-independent decision-makers generalise better than monolithic central planners, for the same reason that ensembles of decision trees generalise better than any single tree. The resource is the diversity of errors. The mechanism is aggregation, and it only works when the errors are genuinely independent.
The three questions any belief model has to answer
Everything so far in this series has been structural. It describes how belief models work, why they fail, and what constrains them. The deeper question is normative. If individuals, organisations, and civilisations are all finite learners fitting bounded models to an unbounded reality, how should they monitor, refine, and update those models?
The question breaks into three sub-questions, each harder than the last.
What are you optimising for?
In machine learning, you choose a loss function before training begins, and the whole apparatus optimises for it. In life, the objective is multi-dimensional (the Yearning Octopus), contested (different stakeholders want different things), and dynamic (what “success” means changes over time). Goodhart’s Law warns that when a measure becomes a target, it ceases to be a good measure.
GDP is the canonical case. It was designed as a descriptive statistic, became a civilisational objective, and now systematically incentivises ecological destruction and inequality because the optimisation pressure has deformed the relationship between the metric and the thing it was supposed to track. The same dynamic plays out across other domains, from education (standardised testing warps curricula) to medicine (fee-for-service warps care) to technology (engagement metrics warp information ecosystems).
For your own belief model, the parallel question is which signals you are weighting as proof that you are right. Weight status, and your model will get good at predicting status. Weight income, and it will get good at predicting income. The model becomes good at producing whatever you reward, and whatever you forget to reward will not be in it.
How do you know it is still fitting reality?
This is the monitoring problem. For an individual it requires calibration, the habit of asking “how often am I right when I’m this confident?” and checking the answer honestly. The forecasting literature has shown that almost no untrained person is calibrated, and that training helps.
For an organisation it requires structured dissent, red teams, pre-mortems, and the discipline of treating near-misses as seriously as actual failures. For a civilisation it requires the institutions that Jonathan Rauch calls “the constitution of knowledge”, namely the norms of empiricism, falsifiability, open inquiry, and institutionalised disagreement that allow large-scale belief systems to error-correct. When monitoring degrades, when calibration turns into arrogance and dissent into disloyalty and empiricism into politics, the belief system loses its ability to detect overfitting until the catastrophic surprise arrives.
How fast should it update?
Too fast and you chase noise. Every new data point triggers a regime change and the system oscillates without ever converging. Too slow and you lock in outdated assumptions. The world has moved, and the model has not noticed. The right update speed depends on the timescale of the underlying change. Individual beliefs should update faster than organisational processes, which should update faster than constitutional structures. When those timescales collapse, the system enters either pathological oscillation (a civilisation rewriting its constitution every election cycle) or pathological rigidity (an individual refusing to update core beliefs across decades of disconfirming evidence). Neither works.
The most robust belief models run on what amounts to a dual-memory architecture. They keep fast, flexible mechanisms for absorbing new information (working memory in individuals, agile teams in organisations, free press and scientific preprints in civilisations) alongside slow, consolidated mechanisms for preserving hard-won structural knowledge (long-term memory, institutional culture, constitutional order). The brain literally implements this. A fast, sparse hippocampal system handles rapid encoding. A slow, distributed cortical system handles gradual consolidation. The same architecture shows up in organisations that pair innovation labs with operational cores, and in civilisations that pair legislative flexibility with constitutional rigidity. Two clocks, deliberately mismatched, so the model can keep pace with the world without losing the deep structure that lets it act in the world at all.
The practice
The bias-variance trade-off started as a theorem about curve-fitting. It has turned out to describe the central constraint on any system that learns from finite experience in a changing world.
The constraint is inescapable. You cannot eliminate bias without raising variance, and you cannot reduce variance without introducing bias. What you can do is choose your operating point deliberately, build regularizers that keep you near the valley, run monitoring systems that tell you when the terrain has shifted, and update your sense of what you are optimising for as your understanding of what matters matures. Regularization, at every scale, is the architecture of resilience.
If Belief Agency made the case that you can choose your beliefs, The Most Dangerous Word made the case that the word belief hides six distinct jobs, and The Lens You See Through made the case that beliefs are the output of a model you cannot directly see, this piece adds the fourth move. The bias-variance trade-off scales. The practice has to scale with it.
At the individual level, the practice means holding your expertise lightly, not as modesty but as regularization. You earned the model that produces your beliefs by living through a particular slice of the world. The model is real, and its outputs are useful. They are also fitted to a slice that is moving. The work is to stay deeply skilled without being captured by your training distribution, to encode the right patterns rather than the surface features of past success, and to welcome the data point that breaks your model, because that is the one telling you the lens has slipped.
At the organisational level, it means building institutions that can exploit and explore at the same time, that reward dissent as much as alignment, and that invest in the absorptive capacity to notice signals the current model cannot parse. At the civilisational level, it means preserving the polycentric, pluralist, empiricist architecture that lets large-scale belief systems learn from their mistakes rather than doubling down on them.
The practical move at every scale is the same. Notice when the model is producing crisp, confident outputs about a regime that may have changed. Notice when it is producing hesitant outputs about a regime where it ought to be confident, because that is the cue that you need richer experience or a different frame. Build the regularizers you do not yet have.
The disconfirming reading. The conversation you have been avoiding. The metric you have been resisting. The practice you have been letting lapse. Build them on purpose.
This is the fourth piece in the Powerful Belief series, following Belief Agency, The Most Dangerous Word, and The Lens You See Through. The argument draws on James March on exploration and exploitation, Joseph Henrich’s cultural-evolution research, Elinor Ostrom on polycentric governance, Jonathan Rauch on the constitution of knowledge, and the dual-memory architecture literature in cognitive neuroscience.