Machine Learning from Scratch

No scikit-learn. Twelve tutorials that implement the core of applied machine learning — gradient descent, ridge and logistic regression, train/test discipline, cross-validation, ROC curves, k-NN, decision trees, k-means, principal components, Gaussian mixtures and random forests — from first principles, in numpy, against real basketball data.

The order is a genuine curriculum: each step uses machinery the previous one built. By the end you will have written, not imported, an entire introductory ML course — and you will know exactly what every library call you make afterwards is doing.

What you'll be able to do

• Implement gradient descent and watch it land on the closed-form answer
• Build logistic regression and evaluate it like a professional: splits, k-fold, ROC/AUC
• Write k-NN, a decision tree and k-means clustering from bare numpy
• Explain regularization by having implemented it
• Find principal components by power iteration, test how many are real, and know what the smallest ones can hide
• Fit a Gaussian mixture with EM, choose its size with BIC, and recognise when the likelihood is rewarding a recording habit rather than structure
• Grow a random forest, read its out-of-bag error, and rank features with a shuffle test rather than the impurity score

Download the course bundle - every script + dataset, one ZIP

The bundle contains each part's finished script, the shared helpers, and the sample data they read, with a README listing the run order - the whole course works offline.

The syllabus

Pace: Strictly sequential through part 6; the last six are siblings you can reorder, as long as the decision tree (part 8) comes before the random forest (part 12). Give it three to four focused weeks.

1. 1Gradient Descent From Scratch: How Models Actually LearnAdvanced · ~5 minThe engine under everything here: follow the gradient downhill.
2. 2Ridge Regression From Scratch: Taming Collinear FeaturesAdvanced · ~5 minRegularization as a dial you implemented, not a checkbox you ticked.
3. 3Logistic Regression From Scratch: Predicting a Win, Not a NumberAdvanced · ~7 minFrom predicting numbers to predicting wins - the sigmoid changes the game.
4. 4Train/Test Split and the Confusion Matrix: Grading a Classifier HonestlyAdvanced · ~6 minThe discipline: never grade a model on the data that taught it.
5. 5K-Fold Cross-Validation: A Score You Can TrustAdvanced · ~5 minOne split can lie; five folds lie less.
6. 6ROC Curves and AUC From ScratchAdvanced · ~6 minThresholds are choices; ROC shows every one of them at once.
7. 7k-Nearest Neighbors From ScratchAdvanced · ~6 minNo training at all - prediction by neighborhood vote.
8. 8A Decision Tree From ScratchAdvanced · ~5 minInterpretable splits, built greedily - see exactly why it predicts.
9. 9K-Means Clustering From Scratch: Find Team ArchetypesAdvanced · ~5 minNo labels needed: let the data cluster itself.
10. 10Principal Component Analysis: Where a Box Score Hides Its ShootingAdvanced · ~28 minNo labels again: how many numbers a box score really holds, and why the smallest one can matter most.
11. 11Gaussian Mixtures: EM Finds the Court's ZonesAdvanced · ~31 minK-means with soft edges: EM draws the court's zones, and a likelihood that can run to infinity.
12. 12Random Forests: A Column of Noise Ranks ThirdAdvanced · ~30 minPart 8's tree, grown 500 times and put to a vote, and an importance score that ranks pure noise.

Check yourself

Five questions, graded on the page, nothing recorded anywhere - pick an answer and the explanation appears.

Where next: Randomness, Inference & Simulation.