Bias–Variance Trade-off

1.8 — Bias–Variance Trade-off

Recall first. Consider a very simple model and a very flexible model trained on different samples from the same population. Which one is more likely to give nearly the same prediction every time? Which one is more likely to chase quirks of the sample?

First, remove the ambiguity around “bias”

In this note, bias means systematic prediction error from a model family that cannot represent the underlying pattern well. It is not:

• the intercept/bias parameter in a linear equation;
• sampling or social bias in a dataset; or
• a personal preference.

Variance means how much the fitted model’s prediction changes when the training sample changes. To discuss variance, imagine repeatedly drawing different training sets from the same population and fitting the same learning procedure to each one.¹²

The squared-error decomposition

For a fixed input x, suppose the target is:

Y = f(x) + ε

where f(x) is the underlying signal and ε is irreducible noise with mean zero. If f̂_D(x) is the model fitted on training set D, then under squared loss:

E[(Y − f̂_D(x))²]
  = noise variance + Bias[f̂_D(x)]² + Variance[f̂_D(x)]

with:

Bias = E_D[f̂_D(x)] − f(x)
Variance = E_D[(f̂_D(x) − E_D[f̂_D(x)])²]

The expectation over D asks what happens across possible training samples. The decomposition explains why lowering training error is not the whole goal: a model can trade lower bias for higher variance.²³

What model complexity changes

A simple model makes similar predictions across datasets, but they may all be wrong in the same direction. A flexible model can fit a wider range of patterns, but its fitted details may change greatly when the sample changes.

prediction error
  ^                 total test error
  |                    \__/
  |  bias²  \          /
  |          \        /
  |           \______/   variance rises →
  +----------------------------------------> complexity

The U-shaped total-error curve is a useful heuristic. Its exact shape depends on the data, model, loss, and training procedure.

Worked numerical intuition

Assume the true mean target at one input is 10. Fit a simple model on three different training samples:

predictions: 8, 8, 8

Its mean prediction is 8: squared bias is (8 − 10)² = 4, and variance is 0 in this toy set.

Now fit a flexible model on three different samples:

predictions: 6, 10, 14

Its mean prediction is 10: squared bias is 0, but its variance is:

((6−10)² + (10−10)² + (14−10)²) / 3 = 32/3 ≈ 10.67

The flexible model is centered correctly in this toy example but is unstable across training samples. More flexibility did not automatically create a better expected prediction.

How to respond to the trade-off

To reduce high variance:

• collect more representative training data;
• simplify the model;
• apply regularization;
• reduce irrelevant features;
• use cross-validation and honest model selection;
• stop training when validation performance stops improving.

To reduce high bias:

• improve or add informative features;
• choose a more expressive model;
• reduce excessive regularization;
• change the representation or objective;
• check whether the target is predictable from the available inputs.

More data often helps variance more than bias. A more flexible model often lowers bias but can raise variance. These are tendencies, not promises; validation data decides what works for the actual problem.¹⁴

Bias–variance is not the same as training vs. test error

Training/test behavior helps diagnose the trade-off, but the concepts are different:

• Training error is measured on one observed sample.
• Bias and variance describe expected behavior across possible training samples at a fixed input.
• Generalization error is expected loss on new examples from the target population.

A validation curve can reveal the practical result of the trade-off, while the decomposition explains one reason the curve can exist.

Exercise

A model’s training and validation errors are both high and close. A second model has very low training error and much higher validation error. Which component is likely dominant in each case, and what is one remedy?

Exam lens

Definition: bias is systematic error of the model family; variance is sensitivity of the fitted model to the training sample.

Formula to remember: under squared loss, expected prediction error = irreducible noise + bias² + variance.

Diagram to reproduce: as complexity rises, bias² tends to fall, variance tends to rise, and total validation/test error often forms a U-shape.

Common traps:

• bias here is not fairness bias or the intercept parameter;
• variance here is not the variance of one feature;
• a high-variance model is not “random” in the ordinary sense—it is unstable across training samples;
• the trade-off is managed by validation, not solved by always choosing the simplest or most complex model.

Rapid revision

• Can I define bias and variance using repeated training samples?
• Can I write the squared-error decomposition?
• Can I explain why a simple model can have high bias and low variance?
• Can I match a remedy to high bias or high variance?

Sources