Word
double descent
Definition
n. a surprising pattern where an AI model's test mistakes go down, spike at a critical size, and then go down again as it gets even larger.
n. the phenomenon in overparameterized models where test risk decreases, peaks near the interpolation threshold where parameters equal sample size, and decreases again.
Examples
“Engineers observed double descent when making the neural network bigger actually improved its accuracy on tests.”
“The discovery of double descent reconciled classical statistical trade-offs with the modern success of massive deep learning models.”
“In modern overparameterized regimes, double descent demonstrates that implicit regularization in gradient descent guides overfitted networks toward minimum-norm solutions.”
Examples
simple
“Engineers observed double descent when making the neural network bigger actually improved its accuracy on tests.”
contextual
“The discovery of double descent reconciled classical statistical trade-offs with the modern success of massive deep learning models.”
complex
“In modern overparameterized regimes, double descent demonstrates that implicit regularization in gradient descent guides overfitted networks toward minimum-norm solutions.”
Real-World Examples
“We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time.” “This “double-descent” curve subsumes the textbook U-shaped bias–variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance.” Etymology
Coined in 2018–2019 by Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal in 'Reconciling modern machine learning and the bias-variance trade-off'.
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.
Domains
Scan code
englishreference.com/q/double-descent