Word
backpropagation
Definition
n. the algorithm a neural network uses to calculate its mistakes and figure out how to adjust its internal weights.
n. an efficient algorithm that applies the mathematical chain rule backwards through computational graphs to compute gradients of the loss with respect to all parameters.
Examples
“During training, backpropagation tells the network how to correct its errors on each picture.”
“The breakthrough in deep learning occurred when backpropagation was successfully scaled on graphics processing units.”
“Vanishing and exploding gradients during backpropagation through deep architectures necessitated the introduction of residual skip connections and layer normalization.”
Examples
simple
“During training, backpropagation tells the network how to correct its errors on each picture.”
contextual
“The breakthrough in deep learning occurred when backpropagation was successfully scaled on graphics processing units.”
complex
“Vanishing and exploding gradients during backpropagation through deep architectures necessitated the introduction of residual skip connections and layer normalization.”
Real-World Examples
“Backpropagation, the topic of this lesson, is an algorithm for computing that negative gradient.” “Backpropagation is the key algorithm that makes training deep models computationally tractable.” “Standard backpropagation learning builds up brittle co-adaptations that work for the training data but do not generalize to unseen data.” Etymology
Short for 'backward propagation of errors', popularized in 1986 by David Rumelhart, Geoffrey Hinton, and Ronald Williams in their landmark paper in Nature.
Etymology adapted from Wiktionary, available under CC BY-SA 4.0.