Word
softmax
Definition
n. a mathematical formula that converts a list of raw numbers into probabilities that add up to one hundred percent.
n. a normalized exponential function that maps an arbitrary K-dimensional vector of real values to a valid probability distribution over K choices.
Examples
“Applying softmax turned the model's raw scores into clear percentage chances for each word.”
“The attention mechanism in transformers uses softmax to weight which tokens to focus on.”
“When logits differ by large margins, the softmax function behaves like a differentiable hard-max, collapsing probability onto the dominant element.”
Examples
simple
“Applying softmax turned the model's raw scores into clear percentage chances for each word.”
contextual
“The attention mechanism in transformers uses softmax to weight which tokens to focus on.”
complex
“When logits differ by large margins, the softmax function behaves like a differentiable hard-max, collapsing probability onto the dominant element.”
Real-World Examples
“In standard attention, which we call soft attention, the attention weights are computed by the softmax function and cannot be exactly 0 (unless there is masking) or exactly 1 (unless there is only one position).” “Softmax converts a list of attention scores into a probability distribution: numbers between 0 and 1 that sum to 1: so the model can decide how much weight to give each word.” Etymology
Coined by British computer scientist John S. Bridle in 1989 for a smooth, continuously differentiable mathematical approximation to the maximum ('arg max') function.
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.
Domains
Scan code
englishreference.com/q/softmax