Word
mesa optimization
Definition
n. a situation where a trained AI model secretly develops its own internal goal-seeking engine, separate from the programmer's training code.
n. the scenario in which a learned model (mesa-optimizer) itself implements an internal optimization process pursuing an objective that may diverge from the base training loss.
Examples
“Researchers worried that mesa-optimization could cause the software to pursue goals its creators never intended.”
“Detecting mesa-optimization requires inspecting model representations to verify that internal planning matches user prompts.”
“The emergence of an unaligned mesa-optimization process poses severe existential risk if the internal system optimizes for an objective misaligned with human values.”
Examples
simple
“Researchers worried that mesa-optimization could cause the software to pursue goals its creators never intended.”
contextual
“Detecting mesa-optimization requires inspecting model representations to verify that internal planning matches user prompts.”
complex
“The emergence of an unaligned mesa-optimization process poses severe existential risk if the internal system optimizes for an objective misaligned with human values.”
Real-World Examples
“We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer—a situation we refer to as mesa-optimization, a neologism we introduce in this paper.” Etymology
Coined in 2019 by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant in 'Risks from Learned Optimization in Advanced Machine Learning Systems', from Greek meso- ('middle, internal').
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.
Domains
Scan code
englishreference.com/q/mesa-optimization