Curriculum

ML Domain · Model Combination

Ensemble Methods

Ensembles combine multiple weak learners into one strong predictor. Bagging reduces variance; Boosting reduces bias. Gradient Boosting and Random Forests dominate tabular ML competitions.

The idea

A single decision tree is fast and interpretable but suffers from high variance—small data changes produce very different trees. Ensembles fix this by averaging many trees.

Bagging (Bootstrap Aggregating) trains each tree on a random bootstrap sample of the data. Random Forests add feature randomisation: each split only considers a random subset of features, decorrelating the trees and reducing variance further.

Boosting is different: models are trained sequentially, each correcting the errors of the previous. Gradient Boosting fits each new tree to the residuals (pseudo-residuals) of the ensemble so far. XGBoost and LightGBM are engineering-optimised versions that dominate Kaggle tabular competitions.

F(x) = Σₘ αₘ · hₘ(x)

Ensemble prediction as a weighted sum of M weak learners hₘ with weights αₘ.

What to hold onto

  • Bias-Variance DecompositionBagging targets variance; boosting targets bias. Knowing which is your bottleneck determines which approach to reach for.
  • Feature ImportanceBoth RF and GBM rank features by how much splits on them reduce loss. These scores guide feature selection and domain insight.
  • StackingTrain a meta-learner on the out-of-fold predictions of a diverse set of base models. Often outperforms any single model.

Live: Random Forest & Gradient Boosting Boundary

Watch how aggregating multiple weak decision trees creates a smooth, robust non-linear decision boundary without overfitting.

Quick answers

What are Ensemble Methods in Machine Learning?

Ensemble Methods combine predictions from multiple models to achieve better generalisation than any individual model. Bagging trains models on bootstrap samples to reduce variance; Boosting trains them sequentially to reduce bias; Stacking learns a meta-model over base-model predictions.

What is the difference between Bagging and Boosting?

Bagging trains models in parallel on random subsets of data and averages their outputs, primarily reducing variance. Boosting trains models sequentially, each correcting the previous model's errors, primarily reducing bias. Random Forests use bagging; XGBoost uses boosting.