Vanishing & Exploding Gradients

Source: Unit 2 §10

When training deep networks, the computed derivatives can become very large (exploding) or very small (vanishing), and either one makes training difficult or impossible.

Why it happens

Consider a deep net with all biases 0 and a linear activation $a(x) = x$. Then the output is simply a product of all the weight matrices:

$$y = W_L \cdot W_{L-1} \cdot \ldots \cdot W_1 \cdot [x]$$

Z1 = W1·x, a1 = a(Z1) = Z1, a2 = a(W2·a1), …, aL = a(WL·aL−1)

A per-layer scale of 1.5 or 0.5 looks harmless. Raised to the power of the depth it is not: by layer 10 the signal is 58× too big or 1000× too small.

#NumbersThe exponential in the depth

• If each weight matrix scales its input by, say, 1.5, then over $L$ layers the factor is $1.5^{L}$, which explodes for large $L$.
• If it scales by 0.5, the factor is $0.5^{L}$, which vanishes exponentially.
• The gradients used in gradient descent behave the same way, so they explode or vanish as a function of the depth $L$.

⚠GotchaVanishing is the quieter failure

Exploding gradients announce themselves - the loss goes to infinity or NaN. Vanishing gradients look like a network that simply is not learning: the early layers receive a gradient of approximately 0, so their weights never move and training silently stalls.

The solution: careful weight initialisation

A partial solution is a more careful choice of random weight initialisation.

For a single neuron computing $z = w_1 x_1 + \ldots + w_n x_n$: the larger $n$ (the number of inputs), the smaller each $w_i$ should be, so that $z$ neither blows up nor collapses. That is achieved by setting the variance of the weights:

$$\mathrm{Var}(w_i) = \frac{1}{n}$$

In code, per layer, where the layer has $n^{[L-1]}$ inputs:

W_L = np.random.randn(shape) * np.sqrt(1 / n[L-1])

Activation	Recommended variance	Name
tanh	1 / n^{[L−1]}	Xavier initialisation
ReLU	2 / n^{[L−1]} (works better)	He initialisation

◎Exam cueWhich initialiser goes with which activation

tanh takes Xavier ($1/n$); ReLU takes He ($2/n$). ReLU zeroes out roughly half its inputs, so it needs twice the variance to keep the same signal scale leaving the layer.