Supervised Models

Four families of supervised learner, in the order the course builds them. A decision tree partitions the feature space with axis-parallel cuts; KNN refuses to build a model at all and defers everything to query time; a neural network learns a continuous function by gradient descent; an SVM looks for the single widest margin and then quietly moves the whole problem into a higher dimension.

Decision trees

• Decision Trees & ID3 - entropy, information gain, and the algorithm that greedily grows a tree.
• Hypothesis Space Search & Inductive Bias - what ID3 is really searching, and what it silently prefers.
• Issues in Decision Tree Learning - overfitting, pruning, continuous attributes, missing values.

Instance-based learning

• Lazy vs Eager Learning - the trade-off between training time and query time.
• K-Nearest Neighbours - the algorithm, the distance metrics, and the choice of k.
• Weighted KNN & Its Problems - distance weighting, and the curse of dimensionality.

Neural networks

• ANN Foundations - the biological analogy and when a network is the right tool.
• The Perceptron - the single unit, its training rule, and what it cannot do.
• Gradient Descent & the Delta Rule - the error surface and how weights move down it.
• Multilayer Networks & Forward Propagation - stacking layers, and the sigmoid unit that makes it work.
• Backpropagation - the chain rule turned into a training algorithm.
• Activation Functions - sigmoid, tanh, ReLU and friends.
• Vanishing & Exploding Gradients - why deep networks stopped training, and what fixed it.

Support vector machines

• Support Vector Machines - maximum margin, support vectors, and the soft margin.
• Kernels & the Kernel Trick - separating the inseparable without ever visiting the higher dimension.
• Cheat Sheet - the whole topic on one page.