Word
grokking
Definition
n. a dramatic breakthrough where an AI model suddenly learns the true underlying rule long after it has already memorized its training data.
n. the phenomenon where an overfitted neural network abruptly transitions from memorization to perfect test generalization long after training loss has plateaued near zero.
Examples
“After training on modular arithmetic for thousands of extra epochs, the model achieved sudden grokking.”
“Weight decay appears to be an essential regularizer for inducing grokking on algorithmic reasoning datasets.”
“Mechanistic audits reveal that grokking occurs when slow weight regularization gradually prunes memorization circuits in favor of cleaner generalizing Fourier representations.”
Examples
simple
“After training on modular arithmetic for thousands of extra epochs, the model achieved sudden grokking.”
contextual
“Weight decay appears to be an essential regularizer for inducing grokking on algorithmic reasoning datasets.”
complex
“Mechanistic audits reveal that grokking occurs when slow weight regularization gradually prunes memorization circuits in favor of cleaner generalizing Fourier representations.”
Real-World Examples
“The OpenAI researchers called the phenomenon ‘grokking,’ a term coined by science-fiction writer Robert A. Heinlein in his 1961 novel Stranger in a Strange Land to describe an understanding of something that is so profound, you essentially merge with it.” “We present a simple neural network that can learn modular arithmetic tasks and exhibits a sudden jump in generalization known as “grokking”.” Etymology
Coined by American science fiction novelist Robert A. Heinlein in his 1961 novel 'Stranger in a Strange Land' (from a Martian word meaning 'to drink, understand deeply, and become one with'). Adopted in machine learning by Alethea Power et al. at OpenAI in 2021.
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.
Domains
Scan code
englishreference.com/q/grokking