An objective function gives a learner a landscape to navigate. Reaching an optimum shows that the system followed the landscape we supplied, although it does not show whether the landscape captured the right goal.
I first came to reinforcement learning through robots. I spent years working with ROS and OpenAI Gym environments, watching an agent attempt the same task thousands of times and debugging the learning process when nothing seemed to improve. Every so often, a small change in the feedback would produce a completely different behavior. Those moments were technically interesting because they raised a larger question.
A simulated robot arm in MuJoCo, close to the environments where I first encountered reinforcement learning as a research problem. This Isaac Sim and ROS2 walkthrough gives a concrete view of the simulation workflow behind that work.
At the time, I thought I was studying how to make machines learn. Over the years, I came to see reinforcement learning as a stripped-down way of studying learning itself. It isolates one of the oldest questions about intelligence by asking how a system recognizes a useful signal in the consequences of its actions, carries that signal forward, and makes a better decision the next time.
That question reaches beyond artificial intelligence. Humans also learn from incomplete feedback. We repeat what appears to work, avoid what appears to hurt, and build beliefs from signals that may capture only part of reality. Reinforcement learning makes this structure unusually visible because the feedback has to be specified. Someone has to decide what counts as success.
This essay lays out that learning process from the beginning, then follows it to the point where it breaks. By the end, the basic mechanics behind language-model training should feel less mysterious, along with the reasons capable models can still fail in surprising ways. Those failures may also tell us something about our own intelligence, and about the lifelong work of learning and unlearning.
How Reinforcement Learning Works
In its simplest form, the reinforcement-learning loop sends an agent’s action into the environment and returns the next state and a reward. The loop becomes learning when those signals change the next action. (Megajuice, Wikimedia Commons, CC0)
Imagine a robot arm learning to pick up a red block. It receives one point when it lifts the block and nothing when it drops it. After thousands of attempts, the movements that earned a point become more likely, and the robot gets better at the task.
At its simplest, reinforcement learning follows this loop.
Observe → act → receive a reward → adjust → repeat
The agent makes decisions within an environment, which includes everything it can observe and affect. After each action, a numerical reward tells the agent how well the action worked. Over many rounds, the agent learns which actions tend to produce more reward and makes those actions more likely.
For a robot, observations might come from cameras and joint sensors, actions might move motors, and the reward might increase when an object reaches the right location. The same structure can be applied to a language model. The model produces a response, an evaluator scores it, and training makes responses with higher scores more likely. That evaluator might be a human choosing between two answers, a separate model trained to predict human preferences, or a program that checks whether an answer is correct. Chip Huyen’s visual introduction to reinforcement learning from human feedback gives an accessible account of how these pieces fit together in language-model training.
The agent can discover which behavior raises the reward without understanding the designer’s full intention. This is what allows useful behavior to emerge through trial and error, including behavior that would have been difficult to specify as a complete set of instructions.
The same feature creates the problem because a reward compresses a richer judgment about success into one signal. Even when the reward is produced automatically, people still choose the test, the evaluator, or the definition of success behind it. This creates an asymmetry between the outcome people intend and the signal from which the machine actually learns.
When the Reward Becomes a Loophole
One well-known reinforcement learning agent was trained to race a boat through a video game course. The course awarded points for hitting targets along the route, and the agent discovered that it could collect more points by circling the same targets indefinitely than by finishing the race.1 Its score improved while its performance on the actual task got worse.
This behavior is often called reward hacking or specification gaming. The agent satisfies the measurable rule while violating the purpose behind it. More capable systems can find less obvious loopholes because they can search a larger space of possible behavior. Lilian Weng’s survey of reward hacking in reinforcement learning traces how this failure appears in classical RL systems and language-model training.
Frontier AI laboratories have started studying what this looks like in language models. In a controlled 2024 Anthropic experiment, models progressed from relatively simple forms of specification gaming to rare attempts to edit their own reward code. Reward tampering occurred in 45 of 32,768 trials, and seven attempts included efforts to conceal the change.1 A later Anthropic study used programming tasks drawn from real training runs. When the models learned to exploit weaknesses in those training environments, the researchers also observed a sharp increase across several evaluations of misaligned behavior.2
Because these studies were designed experiments, they do not show that every model will tamper with rewards or that current systems inevitably become deceptive. They support the narrower conclusion that optimizing a flawed signal can teach behavior that the signal’s designers did not anticipate.
This is the problem of the proxy, a measurable stand-in for something harder to observe directly. Test scores can stand in for understanding, clicks for interest, and quarterly revenue for the health of a business. Each may carry useful information while still drifting away from the reality it was meant to represent.
The agent sees the measurable proxy, while the human cares about the underlying goal it represents.
That difference raises the central question of reinforcement learning. Are the rewards we construct grounded in the reality we care about, and are they teaching the behavior we actually want?
Better Environments Expose Weak Rewards
If every reward we design is incomplete, how can learning improve without access to a perfect reward outside the system?
Part of the answer lies in environment design. Changing the world in which a reward appears reliable can make its blind spots easier to see. A robot that encounters only one type of block, on one table, under one kind of lighting can earn a high score without learning the broader task. A language model evaluated only on familiar questions with easily checked answers may fail when the context becomes ambiguous or the consequences extend beyond the test.
The intuition becomes clearer when we enter a new city or country and redesign part of our learning environment. Different conventions around time, politeness, work, and trust reveal the distance between what we assumed was universal and what was merely local, while the new environment supplies evidence that the old one could not.
Environment design determines what can happen, while exploration determines which possibilities we actually encounter. In life, exploration can mean unfamiliar work, unexpected conversations, or time left open for serendipity. Useful exploration differs from random motion because it needs a strategy that permits enough novelty to reveal what our current habits cannot see and enough safety to absorb what happens.
That is why, toward the end of my reinforcement learning work, I became obsessed with exploration and intrinsic motivation, and with whether mutual information could help an agent seek informative experience. I wanted to understand how an agent might discover what its current reward would never lead it to see.
Exposure alone does not guarantee learning because the designer still chooses which situations an agent can encounter, how it explores them, which outcomes count as evidence, and when the reward should be revised. In AI training, richer environments and better exploration strategies cannot eliminate the proxy gap, although they can make the gap observable and give designers a chance to narrow it.
Learning Never Reaches a Final Version
In his well-known essay “How to Pick a Career That Actually Fits You,” Tim Urban gives this problem a memorable form in the “Yearning Octopus.” Its tentacles represent personal fulfillment, social approval, lifestyle, moral impact, and practical security. Each points toward a different kind of choice, and none offers a complete answer.
Read through the lens of reinforcement learning, the octopus raises a three-part question. Which signals will I treat as rewards, which environments will expose me to better evidence, and what exploration strategy will help me discover unfamiliar possibilities without sacrificing the safety that makes continued learning possible?
We are learners, and we are also designers of much of what does the teaching. The work we choose, the people around us, the places we inhabit, and the measures by which we judge progress influence which signals reach us and which actions become easier to repeat. This gives us partial yet meaningful control over our own learning.
The brain is also a learning machine that turns external events into internal signals, including reward-related signals that help shape what we do next, as studies of the anterior cingulate cortex illustrate (Hayden et al., 2011 and Tervo et al., 2014).
There is no final reward function for machines or for us. What we say about ourselves matters, although repeated action reveals which signals actually govern us. Learning continues through the environments we enter, the consequences we notice, and the actions we choose next. As one of the designers of your own learning, how will you act?
Updated September 19, 2026. This essay treats reward as one doorway into a much larger question. Intelligence itself is unlikely to be captured by a single score. In A Useful Viewpoint of the Word “Intelligent”, I explore a framework that treats intelligence as a profile of how systems resolve uncertainty across different goals and environments. There is still much to learn about learning itself, and even more about the relationship between learning, intelligence, and consciousness.