Word
superposition
Phoneticssu·per·po·si·tion
Definition
n. the ability of a neural network to represent more concepts than it has physical dimensions by overlapping them.
n. the phenomenon in neural networks where more semantic features are represented than available vector dimensions by exploiting nearly orthogonal directions in high-dimensional space.
Examples
“Quantum particles exist in a superposition of multiple states until measured.”
“Mechanistic interpretability researchers discovered that transformers use superposition to compress thousands of rare concepts into few neurons.”
“Because natural language features are sparse, linear superposition allows networks to store an overcomplete set of features while incurring minimal interference noise.”
Examples
simple
“Quantum particles exist in a superposition of multiple states until measured.”
contextual
“Mechanistic interpretability researchers discovered that transformers use superposition to compress thousands of rare concepts into few neurons.”
complex
“Because natural language features are sparse, linear superposition allows networks to store an overcomplete set of features while incurring minimal interference noise.”
Real-World Examples
“In a recent paper, we found that simple neural networks trained on toy tasks often exhibit a phenomenon called superposition, where they represent more features than they have neurons.” “Neural networks often pack many unrelated concepts into a single neuron – a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. This paper provides a toy model where polysemanticity can be fully understood, arising as a result of models storing additional sparse features in "superposition.” Etymology
From French superposition, from superposer ('to place over, overlap'), from Latin super- ('above') + ponere ('to place'). Adopted from quantum physics into mechanistic interpretability.
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.
Domains
Scan code
englishreference.com/q/superposition