Abstractly speaking, the purpose of backpropagation is to train a neural network to make better predictions through supervised learning. More fundamentally, the goal of backpropagation is to determine how model weights and biases should be adjusted to minimize error as measured by a “loss function”. For now, we’ll focus on the output unit representing the correct prediction, which we’ll call Lc. Lc’s activation function is a composite function, containing the many nested activation functions of the entire neural network from the input layer to the output layer.
Hidden Unit Error
We use the color for data/activations being passed forward through the network. All these arrays represent the gradient at a single operating point, namely that of the current value of the data and parameters. We will use the color to indicate free parameters, which are set via learning and are not the result of any other processing.
That equation for z is therefore part of the activation functions in the next layer and, by extension, also part of every activation function for any neurons in any subsequent layer. Neurons in the “input layer” receive input data, usually as a vector embedding, with each input neuron receiving an individual feature of the input vector. For example, a model that works with 10×10 pixel grayscale images will typically have 100 neurons in its input layer, with each input neuron corresponding to an individual pixel.
9 Backpropagation to the Data
The activation function through its derivative plays a crucial role in computing these gradients during Back Propagation. Another consideration in gradient descent is how often to update weights. One option is to compute the gradients for every example in the training data set, then take an average of those gradients and use it to update parameters.
Vanishing/Exploding Gradient Problem
Neural networks that have more than one layer, such as multilayer perceptrons (MLPs), on the other hand, must be trained using methods that can change the weights and biases in the hidden layers as well. Another interesting property, which we already pointed out previously, is that the backward network only consists of linear layers. This is true no matter what the forward network consists of (even if it is not a conventional neural network but some arbitrary computation graph). The reason why this happens is because backprop implements the chain rule, and the chain rule is always a product of Jacobian matrices. But more intuitively, you can think of each Jacobian as being a locally linear approximation to the loss surface; hence each can be represented with a linear layer. Modern deep neural networks, often with dozens of hidden layers each containing many neurons, might comprise thousands, millions or—in the case of most large language models (LLMs)—billions of such adjustable parameters.
5 Backpropagation Over Data Batches
We start at the error node and move back one node at a time taking the partial derivative of the current node with respect to the node in the preceding layer. Each term is chained onto the preceding term to get the total effect, this is of course the chain rule. There are a few interesting things about this forward-backward network. One is that activations from the relu layers get transformed to become parameters of a linear layer of the backward network (see Equation 14.7).
Instead, there’s a clever trick for speeding up the process in a way that involves more total gradient descent steps but much less time for each one. The final equations in matrix notation are shown in figure 6 where I’ve used capital letters to denote the matrix/vector form of the variable. The first two terms on the right hand side are re-factored to a delta term. As we trace the preceding layers out these terms will occur repeatedly so it makes sense to calculate them once and store it as the delta variable for future use.
Defining Neural Network
So you really could build something amazing without knowing a thing about it. In the same vain you could also drive your car, as many of us do, without the faintest idea of how an engine really works. You don’t have to think about it and can still drive without issue…that is until you’re broken down on the side of the road. Then you sure wish you understood something about what’s going on under the hood. This book may not be used in the training of large language models or otherwise be ingested into large language models or generative AI offerings without OpenStax’s permission.
Visualizations like this are a useful way to figure out what visual features a given neuron is sensitive to. Researchers often combine backpropagation tutorial this visualization method with a natural image prior in order to find an image that not only strongly activates the neuron in question but also looks like a natural photograph (e.g., 3). In Chapter 25, we will see that neural nets can also include cycles and still be trained with variants of backprop (e.g., backprop through time). IBM® Granite™ is our family of open, performant and trusted AI models, tailored for business and optimized to scale your AI applications. The sigmoid function returns a value between 0 and 1, introducing non-linearity into the model. In some sense, the neurons that fire while seeing a 2 get more strongly linked to those firing while thinking about a 2.
Consider the simplest case of a single feedback loop from one neuron to itself. The effect of the connecting weight is equivalent to connecting to a copy of the same RNN (with identical weights and biases). Of course, since the second copy of the RNN has the same feedback loop, it can be unrolled to a third, fourth, fifth copy, etc. The effect of feeding the entire vector x(0)x(0) into the unrolled model is equivalent to feeding the data points (x1,x2,x3,…,xn)(x1,x2,x3,…,xn) sequentially into the RNN. In the simplest such model, one feedback loop connects a single neuron with itself.
- Doing this for all your tens of thousands of training examples, and averaging all the results, gives you the total cost of the network.
- The effect of feeding the entire vector x(0)x(0) into the unrolled model is equivalent to feeding the data points (x1,x2,x3,…,xn)(x1,x2,x3,…,xn) sequentially into the RNN.
- To do so requires training on a large number of samples that reflect the diversity and range of inputs the model will be tasked with making predictions on post-training.
Machine Learning Practice
It works by propagating errors backward through the network, using the chain rule of calculus to compute gradients and then iteratively updating the weights and biases. Combined with optimization techniques like gradient descent, backpropagation enables the model to reduce loss across epochs and effectively learn complex patterns from data. In this post we calculated the backpropagation algorithm for some simplified examples in detail. The general concept of calculating the gradient is calculating the partial derivatives of the loss function using the chain rule. The contributions Lc receives from L-1 neurons are determined not just by the weights applied to L-1’s output values, but by the actual (pre-weight) output values themselves.
- In the remaining sections, we will still focus only on the case of backpropagation for the loss at a single datapoint.
- On the other hand, if connecting weights are too small to begin with, then training can cause them to quickly approach zero, which is called the vanishing gradient problem.
- In the backward pass, we want to update all the four model parameters – the two weights and the two biases.
- One way to change Lc’s output is to change the weights between the neurons in L-1 and Lc.
If you’re beginning with neural networks and/or need a refresher on forward propagation, activation functions and the like see the 3B1B video in