Maximum Likelihood & Least Squared Error

Source: Unit 3 §7

The result

◎Exam cueWhy squared error is not an arbitrary choice

Under certain assumptions, any learner that minimises the squared error between its predictions and the training data outputs the maximum likelihood hypothesis $h_{\text{ML}}$.

This has already happened repeatedly without being named: neural networks, linear regression and polynomial curve fitting all minimise the sum of squared errors. Each of them is, silently, doing maximum likelihood estimation under a Gaussian noise model.

The derivation

Setup. A learner $L: X \to \mathbb{R}$ learns a target $f: X \to \mathbb{R}$. The training data is noisy: each example $\langle x_i, d_i \rangle$ satisfies

$$d_i = f(x_i) + e_i$$

where $e_i$ is Gaussian noise with mean 0. Assume all hypotheses are equally probable a priori, so MAP reduces to ML. Because $e_i$ is Normal, $d_i$ is Gaussian with mean $\mu = h(x_i)$.

Each observation is the true value plus a zero-mean Gaussian draw. The density of that bell at the observed point IS the likelihood of the example, and a Gaussian density has the squared residual sitting in its exponent.

→StepsFrom likelihood to least squares

1. Start from the definition and use independence across examples: $h_{\text{ML}} = \arg\max_h P(D \mid h) = \arg\max_h \prod_i p(d_i \mid h)$, a product of Gaussian densities.
2. Take the log. Maximising $\ln$ of a product is maximising the sum of logs, and $\ln$ is monotone so the argmax is unchanged. This is the standard ML trick.
3. Drop the constant term - the Gaussian's normalising factor is independent of $h$ and cannot change the argmax.
4. What is left is a negative quantity: minus the sum of squared residuals over $2\sigma^2$. Maximising a negative quantity is minimising the positive one, which is the exponent's squared term.

$$h_{\text{ML}} = \arg\min_{h} \sum_i \bigl(d_i - h(x_i)\bigr)^2$$

That is exactly the least-squared-error hypothesis.

≡FactsThe classification counterpart

The same argument run with a different noise model gives a different loss. The ML hypothesis for predicting probabilities is the one that minimises cross-entropy loss. Squared error and cross-entropy are the same idea under two different assumptions about how the data was generated.

⚠GotchaThe assumptions are doing real work

The equivalence needs zero-mean Gaussian noise on the target values only and a uniform prior over hypotheses. Noise on the inputs $x_i$, or a non-Gaussian error distribution, breaks the derivation, and squared error stops being the maximum likelihood objective.