Backpropagation

Source: Unit 2 §8

Backpropagation iteratively compares the network's prediction to the actual target and modifies the weights to minimise the loss - working backwards from the output layer, through each hidden layer, to the first hidden layer.

Two passes over the same wires: values travel forwards to make a prediction, blame travels backwards to change the weights.

≡FactsThe shape of one training step

• For each training tuple, the weights are modified to minimise the loss between the prediction and the target.
• The modifications flow output → hidden → … → first hidden, which is where the "back" in backpropagation comes from.

The loss function

For a single output unit, the error as a function of the weight vector is:

$$E(\vec{w}) = \frac{1}{2} \sum_d (t_d - o_d)^2$$

For multiple output units $k$, sum over the outputs as well:

$$E(\vec{w}) = \frac{1}{2} \sum_d \sum_k (t_{kd} - o_{kd})^2$$

The stochastic approximation updates per example rather than summing over the whole training set first.

The chain-rule derivation

Take a simple chain of single neurons and look at the last layer $L$. We want to know how sensitive the cost $E$ is to each weight, that is $\partial E / \partial w^{(L)}$. A tiny nudge in $w^{(L)}$ nudges $z^{(L)}$, which nudges $a^{(L)}$, which changes $E$. Chain them together:

$$\frac{\partial E}{\partial w^{(L)}} = \frac{\partial z^{(L)}}{\partial w^{(L)}} \cdot \frac{\partial a^{(L)}}{\partial z^{(L)}} \cdot \frac{\partial E}{\partial a^{(L)}}$$

Each arrow owns one derivative. Multiplying the three of them is the entire chain rule for the last layer.

Term	Comes from	Value
∂E/∂a(L)	E = (a(L) − y)²	**2(a(L) − y)**
∂a(L)/∂z(L)	a(L) = σ(z(L))	**σ'(z(L))**
∂z(L)/∂w(L)	z = w·a(L−1) + b	**a(L−1)**

Multiplying the three:

$$\frac{\partial E}{\partial w^{(L)}} = a^{(L-1)} \cdot \sigma'(z^{(L)}) \cdot 2(a^{(L)} - y)$$

≡FactsThree consequences of that one line

• The bias derivative is almost identical. Replace $\partial z / \partial w$ with $\partial z / \partial b$, which is 1, giving $\partial E / \partial b^{(L)} = \sigma'(z^{(L)}) \cdot 2(a^{(L)} - y)$.
• Generalise over $n$ examples by averaging the per-example gradients.
• Keep iterating the chain rule backwards to get the sensitivity to the earlier weights $w^{(L-1)}$, $w^{(L-2)}$ and so on.

Full multi-neuron notation

With many neurons per layer, the indices arrive:

≡FactsReading the superscripts and subscripts

• $a_j^{(L)}$ is the activation of the j-th neuron in layer $L$; $a_k^{(L-1)}$ is the k-th neuron in layer $L-1$.
• $W_{jk}^{L}$ is the weight on the edge connecting the k-th neuron in layer $L-1$ to the j-th neuron in layer $L$.
• The cost for a multi-output layer is $E = \sum_j (a_j^{(L)} - y_j)^2$.

One earlier neuron feeds every later neuron, so its share of the blame is a sum over all the paths it took to reach the cost.

⚠GotchaA neuron influences the cost through more than one path

A neuron in layer $L-1$ feeds every neuron in layer $L$, so it affects the cost by several routes at once. Its $\partial E / \partial a$ is therefore a sum over those paths, not a single term. Forgetting the sum is the classic backprop derivation error.

These chain-rule expressions give the derivatives that form each component of the gradient, which is then used to minimise the cost by repeatedly stepping downhill - ordinary gradient descent, applied to every weight in the network at once.

Derivative of the sigmoid

Backpropagation needs $\sigma'$ at every layer, and the sigmoid has a particularly convenient one:

$$\sigma(x) = \frac{1}{1 + e^{-x}} \qquad \Longrightarrow \qquad \sigma'(x) = \sigma(x)\,(1 - \sigma(x))$$

Derived with the quotient rule. The same exercise for $\tanh$ gives:

$$\tanh'(x) = 1 - \tanh^2(x)$$

◎Exam cueBoth derivatives are written in terms of the function itself

$\sigma' = \sigma(1-\sigma)$ and $\tanh' = 1 - \tanh^2$. During the backward pass the forward activations are already stored, so neither derivative costs a fresh exponential.

Overfitting in backpropagation

The situation is analogous to the decision-tree case. With many weights and many iterations, backpropagation can overfit - tuning the weights to idiosyncrasies of the training data that are not representative of the general distribution.

Training error never stops falling, so it cannot tell you when to stop. The validation minimum can, and the weights at that point are the ones to keep.

→StepsThe validation-set fix

1. Maintain a separate validation set, apart from both the training and the test sets.
2. Periodically measure accuracy on the validation set while training continues.
3. Choose the weights that gave the least validation-set error, not the ones training ended on.