Title here
Summary here
Module · 5 lessons
Boosting Foundations
How boosting differs from a single tree and from random forests, how gradient boosting fits residuals for regression and classification, and the loss-and-gradient idea behind every boosting library.
At a glance
Level
Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Boosting Foundations, the first module of the Gradient Boosting & XGBoost course. Before you touch XGBoost, you’ll build a rock-solid understanding of what gradient boosting actually does. You’ll see why boosting — training trees one after another, each fixing the last one’s mistakes — differs from the parallel averaging of a random forest, and why it so often wins on tabular data.
You’ll learn the mechanics step by step: the additive model that keeps adding shrunken trees to fit the leftover residuals, how the same idea works for classification in log-odds space, and the unifying principle that ties it all together — every tree fits the negative gradient of a loss function. The module ends with a guided project where you build a complete gradient boosting regressor from scratch in NumPy and scikit-learn, then check it against the real thing.
Every model here is trained for real on the California Housing and Adult Income datasets. Start with Lesson 1, where you’ll see exactly why a crowd of weak trees beats a single strong one.
Lessons in this module
1
From Trees to Boosting
See why a single decision tree overfits, how random forests fix it with bagging, and how boosting takes a different path by training trees sequentially on the California Housing dataset.
2
How Gradient Boosting Works
Build a gradient booster from scratch for regression on the real California Housing data, fitting each tree to the residuals of the last and cross-checking against scikit-learn
3
Gradient Boosting for Classification
Extend gradient boosting from regression to classification, predicting who earns more than 50K on the real Adult Income dataset using log-odds, the sigmoid, and log loss.
4
Loss Functions and Pseudo-Residuals
See why gradient boosting fits each tree to the negative gradient of the loss, computing squared-error, absolute-error, and log-loss pseudo-residuals in numpy on real data
5
Guided Project: Build a Gradient Booster from Scratch
Build a working gradient boosting regressor from scratch as a reusable Python class and validate it against scikit-learn on the real California Housing dataset.
Achievement
Complete all 5 lessons to finish the Boosting Foundations module.
