Ensemble Learning Foundations

Source: Unit 3 §1

An ensemble method combines predictions from multiple ML models to make more accurate predictions than any individual model.

The individual learners are usually weak learners - their accuracy is only slightly better than chance, a bit over 50% and not significantly higher. Ensembles are among the most powerful techniques in machine learning, often outperforming everything else, at the cost of increased algorithmic and model complexity.

No single member is good. The combiner is what turns three barely-better-than-chance opinions into one strong prediction.

The two key ideas

≡FactsKey idea 1 - build diverse learners

Multiple learners can be made different from one another via:

• Different algorithms.
• Different hyperparameters of the same algorithm.
• Different subsets of the training data.
• Different features of the training data.

≡FactsKey idea 2 - combine them

Construct multiple diverse models from adapted versions of the data - usually reweighted or resampled - then combine their predictions by simple or weighted averaging or voting.

Why ensembles work: variance reduction

Given $n$ independent observations $Z_1, \ldots, Z_n$, each with variance $\sigma^2$, the variance of their mean is:

$$\operatorname{Var}(\bar{Z}) = \frac{\sigma^2}{n}$$

Averaging reduces variance. That single fact is the whole justification.

≡FactsWhat the variance argument buys you

• Decision trees tend to overfit, which means high variance. Pruning helps but is hard to get right.
• Ensembles combine several weak learners into a final model with low variance.
• Even if the individual learners have high bias, the combined learner can have low bias - and its hypothesis may not even lie in the hypothesis space of any individual learner.

Intuition: independent errors are essential

Picture the instance space as a box, and each learner's errors as a region inside it.

The vote survives an error only when the other learners are not wrong in the same place. Overlapping error regions destroy the ensemble.

⚠GotchaThe requirement that makes or breaks an ensemble

The errors made by the learners must be INDEPENDENT. If the error regions overlap on a point, every learner is wrong there at once and the vote is wrong with them. Independence is achieved by using different subsets of the data or different learners.

Bias-variance recap

	High Bias	High Variance
Model	Too simple, few predictors	Too complex, many predictors
Behaviour	Can't capture the pattern; oversimplifies	Fits training data too well; poor generalization
Error	High error on train AND test	Low train error, high test error
Examples	-	Decision Trees / ANNs (low bias, high variance)

≡FactsReading the trade-off

• More flexible, more powerful representations (low bias) lead to high variance: different data subsets produce different models.
• The goal is low bias and low variance - which is exactly what ensembles achieve.
• Basic models perform poorly because of either high bias (low degrees of freedom) or too much variance (high degrees of freedom).

Confidence from many weak learners

#NumbersHow many votes make you confident

• Ensembles prevent overfitting, so there is no need to worry about a stopping criterion.
• If $n$ learners each have accuracy 0.7 and all predict the same class, confidence in that class is very high.
• In reality $n_1$ learners predict class 1 and $n_2$ predict class 2. Say $n_1 > n_2$, so voting gives class 1. The probability that the true class really is class 1 can be computed from the learners' accuracies, binomial-style.

Combining, and the three types of ensemble

⇄CompareUnweighted vs weighted combination

UnweightedA simple vote or a plain average. Every learner counts the same.

WeightedEach learner's weight is proportional to its **accuracy**, or to **1 / variance**. Better learners pull harder on the result.

Strategy	Examples
Manipulate the data distribution	Bagging, Boosting
Manipulate the input features	Random Forests
Manipulate the class labels	Error-Correcting Output Coding

◎Exam cuePlacing an ensemble method in the exam

Ask what is being manipulated. Resampled or reweighted rows means bagging or boosting; sampled columns means random forests; recoded labels means error-correcting output coding.