What is backpropagation?
Backpropagation is the procedure that works out, for every weight in a neural network, how a small change to that weight would change the network’s error. It runs after a forward pass has produced predictions and the loss function has scored them. Starting at the output, it moves backward layer by layer, applying the chain rule from calculus to pass the error signal to each earlier layer. The result is the gradient that gradient descent uses to update the weights.
Backpropagation is efficient because it reuses intermediate results instead of recomputing the effect of every weight separately. Without it, training a network with billions of parameters would be computationally hopeless. Deep learning frameworks such as PyTorch and TensorFlow perform it automatically, so engineers define the network and the loss and the framework handles the rest.
Why backpropagation matters
For a business reader, backpropagation explains two facts about neural networks. First, training costs roughly three times the compute of inference per example, because the backward pass is about twice as expensive as the forward pass. That is why training needs clusters of graphics processing units (GPUs) while serving the finished model may need a single card. Second, every weight is adjusted a little on every step, so the model’s behavior is a product of all its training data taken together, and no single example can be pointed to as the reason for a given prediction. That is the root of the interpretability problem that explainable AI methods try to address.