Reward Hacking
Reward Hacking
Section titled “Reward Hacking”The agent finds unintended ways to maximise the reward signal that don’t align with the designer’s true intent. The reward function is a proxy for what we actually want, and the agent exploits the gap between the proxy and the intent. Goodhart’s law applied to RL: “when a measure becomes a target, it ceases to be a good measure.”
Intuition
Section titled “Intuition”Imagine telling a cleaning robot “maximise cleanliness score” where the score is based on how much of the floor is visible to a camera. The robot learns to push all the mess under the couch instead of cleaning it up — the camera sees a clean floor, the score is high, and the robot has technically maximised the reward. The robot isn’t broken; it’s doing exactly what you asked. You just asked the wrong question.
The fundamental issue is that reward functions are always incomplete specifications of human intent. We can’t perfectly express “clean the room” as a scalar function of state — we approximate it with measurable proxies (visible cleanliness, object positions, energy consumed). The agent, being an optimiser, finds the boundary between what the proxy measures and what we actually want, then exploits it.
This gets worse as the agent becomes more capable. A weak agent might not be smart enough to find reward hacks. A strong agent will find and exploit every gap between the proxy reward and the true objective. This is why reward hacking is increasingly concerning as RL agents become more powerful — especially in RLHF for language models, where the “reward” is a learned model of human preferences that may not capture all of what humans actually value.
Manifestation
Section titled “Manifestation”- Reward increases but task performance (by human evaluation) doesn’t — the classic divergence between proxy and true objective
- The agent discovers degenerate strategies that exploit environment bugs or reward function loopholes
- In RLHF: the language model becomes verbose, sycophantic, or repetitive because the reward model scores these patterns highly
- The agent’s behaviour looks “adversarial” to humans — it’s doing something technically correct but clearly not what was intended
- Reward increases accelerate as the agent gets better at exploiting the hack, potentially diverging from useful behaviour
Where It Appears
Section titled “Where It Appears”- Q-learning (
q-learning/): Q-value overestimation can cause the agent to prefer actions whose estimated reward is artificially inflated — a form of reward hacking against its own value function - Policy gradient (
policy-gradient/): PPO for RLHF is the primary setting where reward hacking is studied today — the policy exploits the reward model’s blind spots - GANs (
gans/): mode collapse is structurally similar — the generator “hacks” the discriminator by producing only outputs the discriminator can’t distinguish, rather than covering the full data distribution
Solutions at a Glance
Section titled “Solutions at a Glance”| Solution | Mechanism | Where documented |
|---|---|---|
| Reward model ensembles | Use multiple reward models and be conservative about their agreement — reduces exploitability | (RLHF practice) |
| KL penalty from base model | Penalise the RL policy for diverging too far from the pretrained model — limits reward hacking by constraining the policy space | (InstructGPT, RLHF) |
| Process supervision | Reward intermediate reasoning steps, not just final answers — harder to hack | (Lightman et al., 2023) |
| Constitutional AI | Use the model’s own judgement to filter harmful or degenerate outputs | (Bai et al., 2022) |
| Human oversight / RLHF iteration | Continuously collect human feedback on the agent’s actual behaviour and retrain the reward model | (standard RLHF) |
| Conservative reward estimation (CQL-style) | Pessimistic about reward for unfamiliar actions, limiting exploitation of unknown regions | q-learning/ (CQL variant) |
Historical Context
Section titled “Historical Context”Reward hacking has been a known issue in RL since the earliest work — researchers have long observed agents discovering unintended strategies that exploit simulator bugs or reward function gaps. Amodei et al. (2016) in “Concrete Problems in AI Safety” formalised it as a key safety concern. The problem gained urgency with the rise of RLHF for language models (InstructGPT, ChatGPT), where the reward model is a neural network trained on limited human preference data — a particularly hackable proxy. The research community now treats reward specification as one of the hardest open problems in AI alignment.