Boosting

3.3 — Boosting

Recall first. Bagging fits models independently. What information does boosting pass from one learner to the next?

Sequential additive correction

Boosting builds an ensemble sequentially. Each new weak learner is fitted to emphasize what the current ensemble predicts poorly. A generic additive form is:

Fₘ(x) = Fₘ₋₁(x) + η αₘ hₘ(x)

where hₘ is the new learner, αₘ its weight, and η the learning rate/shrinkage. The final prediction is the sum (or a transformed sum for classification). Unlike bagging, the learners are not independent and cannot be freely trained in parallel in the basic algorithm.¹²

Two exam-level views

AdaBoost: start with equal example weights. Fit a weak classifier, increase weights on misclassified examples, decrease or relatively reduce weights on correctly classified examples, and give a learner with lower weighted error a larger vote. For binary y∈{−1,+1}:

αₘ = 1/2 log((1−errₘ)/errₘ)

The weighted combination is sign(Σ αₘhₘ(x)). A learner with error below 0.5 gets positive weight; error 0.5 gets zero weight under the idealized formula. This is an exam simplification; implementations have safeguards and details.

Gradient boosting: start with a baseline prediction and repeatedly fit a learner to the negative gradient (for squared error, residuals) of the chosen loss. Add a shrunken update. Thus “correct previous residuals” is useful shorthand, but the precise target is the loss gradient, not always raw residuals.¹

Worked AdaBoost trace

Suppose a weak classifier has weighted error err=0.20:

α = 0.5 log(0.8/0.2) = 0.5 log 4 ≈ 0.693

If another has error 0.40, its weight is 0.5 log(0.6/0.4)≈0.203; the better learner gets a larger vote. After the first learner, examples it misclassified receive increased relative weight, so the next learner focuses on them. A single bad example may eventually receive very high weight, which explains both the corrective power and sensitivity to label noise/outliers.

Controls and trade-offs

Boosting can reduce bias and produce strong accuracy from shallow trees, but too many stages, large learners, high learning rate, or noisy labels can overfit. Control it with number of estimators, learning rate, tree depth/leaf size, subsampling, regularization, and early stopping based on validation performance. Small learning rates often need more stages; this is a compute/fit trade-off, not a free improvement.¹

Exercise

A binary AdaBoost round has weighted error 0.5. What is its vote weight? If error is 0.1, is the weight positive or negative?

Exam lens

Contrast “sequentially focus on errors” with bagging’s independent resamples. Give AdaBoost’s weight formula or gradient-descent-on-loss view, then state learning rate, stages, and overfitting controls.

Rapid revision checklist

• Can I explain what changes between boosting rounds?
• Can I calculate AdaBoost α from weighted error?
• Can I distinguish residual fitting from the general loss-gradient view?
• Can I name three boosting hyperparameters?

Key takeaways

• Boosting is a sequential additive ensemble of weak learners.
• AdaBoost reweights examples; gradient boosting fits loss gradients.
• Shrinkage and shallow learners control the bias–variance trade-off.
• Boosting can be powerful but sensitive to noise and overfitting.

Sources