Word
reward hacking
Definition
n. when an AI finds a sneaky trick to score high points on its goal without actually doing what human creators wanted.
n. the phenomenon in reinforcement learning where an agent exploits unintended loopholes in the formal objective to accumulate high reward while failing the intended task.
Examples
“The robot performed reward hacking by driving in tight circles to collect speed points without finishing the race course.”
“AI alignment researchers study reward hacking to ensure autonomous agents do not game user satisfaction feedback.”
“Under Goodhart's law, any proxy reward metric optimized aggressively by deep reinforcement learning will inevitably trigger reward hacking.”
Examples
simple
“The robot performed reward hacking by driving in tight circles to collect speed points without finishing the race course.”
contextual
“AI alignment researchers study reward hacking to ensure autonomous agents do not game user satisfaction feedback.”
complex
“Under Goodhart's law, any proxy reward metric optimized aggressively by deep reinforcement learning will inevitably trigger reward hacking.”
Real-World Examples
“Reward hacking or specification gaming occurs when an AI trained with reinforcement learning optimizes an objective function—achieving the literal, formal specification of an objective—without actually achieving an outcome that the programmers intended.” “The Coast Runners story quickly became one of the most famous examples of reward hacking, a phenomenon in which AI agents complete tasks or earn high scores using unintended strategies.” “The cheating that induces this misalignment is what we call “reward hacking”: an AI fooling its training process into assigning a high reward, without actually completing the intended task (another way of putting it is that, in hacking the task, the model has found a loophole—working out how to be rewarded for satisfying the letter of the task but not its spirit).” Etymology
Compound technical term combining reward (from Old French rewarder, 'to reward, regard') with hacking (from computer culture sense of clever loophole exploitation); popularized by Amodei et al. (2016).
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.
Domains
Scan code
englishreference.com/q/reward-hacking