Activation Functions

Source: Unit 2 §9

Activation functions exist to inject non-linearity - without one, a layer is just linear regression and depth buys nothing. Different tasks want different activations, and the five below are the ones the course names.

Sigmoid

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

Sigmoid squashes the whole real line into (0, 1) and never decreases, so its output can be read straight off as a probability.

≡FactsWhat sigmoid gives you

• It is monotonic - entirely non-decreasing.
• The output lies in $(0,1)$, so it can be interpreted as a probability.

Tanh (hyperbolic tangent)

$$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}$$

Tanh is the zero-centred sibling of sigmoid: a negative input comes out strongly negative instead of merely close to zero.

≡FactsWhy tanh is often preferred to sigmoid

• It is monotonic.
• Negative inputs map strongly negative and zero maps to near zero, so the function is zero-centred - which often trains better than sigmoid.

ReLU (rectified linear unit)

$$\mathrm{ReLU}(x) = \max(0, x)$$

ReLU is piecewise linear: it passes positives through untouched and flattens every negative to exactly zero.

≡FactsThe workhorse activation

• One of the most commonly used activations. It converts any negative input to 0.
• Cheap to compute and cheap to differentiate, which matters at every layer of a deep network.

⚠GotchaThe dying ReLU problem

A neuron whose inputs keep landing in the negative region always outputs 0 - and its gradient there is also 0, so no update can ever move it back. It has died and stops learning permanently.

Leaky ReLU

$$\mathrm{LeakyReLU}(x) = \begin{cases} x & x > 0 \\ \alpha x & x \le 0 \end{cases} \qquad \alpha \text{ small, e.g. } 0.01$$

A small negative slope instead of a flat zero. The slope is drawn steeper than a real α = 0.01 so it is visible; the point is only that it is non-zero.

≡FactsThe fix for dying ReLU

• Leaky ReLU is a variant of ReLU that solves the dying-ReLU problem by giving negatives a small slope rather than zero.
• $\alpha$ can be a fixed small value, or a learnable parameter the network tunes for itself.

Softmax

$$\mathrm{softmax}(x_i) = \frac{e^{x_i}}{\sum_j e^{x_j}}$$

≡FactsWhat softmax is for

• It is a generalisation of sigmoid, used in the final layer for multi-class classification.
• It gives the probability of each class, and the outputs sum to 1.

Worked example

Output-layer sums $x = [1.78,\ 3.67,\ 0.33,\ 5.77]$.

→StepsTurning four scores into four probabilities

1. Exponentiate each: $e^{1.78} = 5.93$, $e^{3.67} = 39.25$, $e^{0.33} = 1.39$, $e^{5.77} = 320.5$.
2. Sum them: $\sum_j e^{x_j} \approx 367$.
3. Divide each exponential by the sum: $[0.016,\ 0.106,\ 0.0037,\ 0.8731]$.
4. Check: the four values sum to 1. Class 4 is the most likely, with 87% of the mass.

Exponentiating first is what makes the winner dominate: 5.77 is only 1.6× the next-largest input, but it takes 87% of the probability.

Quick comparison

Function	Range	Note
Sigmoid 1/(1+e⁻ˣ)	(0, 1)	probability; binary output
Tanh	(−1, 1)	zero-centred
ReLU max(0, x)	[0, ∞)	fast, common; can "die"
Leaky ReLU (αx for x < 0)	(−∞, ∞)	fixes dying ReLU
Softmax eˣⁱ/Σeˣʲ	(0, 1), sums to 1	multi-class output layer