Training Error and Generalization Error
1.6 — Training Error and Generalization Error
Recall first. If a model scores perfectly on the examples used to fit it, what can you conclude about its performance on tomorrow’s examples? Write the strongest conclusion you think is justified.
Error is always tied to a loss and a distribution
An error number has meaning only after we specify:
- the task and target;
- the loss function;
- the examples on which the loss is measured; and
- the distribution of examples we care about.
For a dataset D = {(xᵢ, yᵢ)} and a fitted model f̂, let L(y, f̂(x)) be the loss.
Training error
The training error (empirical risk) is the average loss on the examples used to fit the model:
R_train(f̂) = (1/n) Σ L(yᵢ, f̂(xᵢ))It tells us how well the model fits the observed training sample. A flexible model can make this very small by learning details or noise specific to that sample.
Generalization error
The generalization error is the expected loss on a new example drawn from the target population:
R_gen(f̂) = E_(X,Y)~P [ L(Y, f̂(X)) ]The distribution P represents the conditions in which the model will actually be used. We usually cannot calculate this expectation exactly, so a properly held-out test set is used as an estimate.12
Common losses
- Regression: MAE,
|y − ŷ|, treats errors linearly; MSE,(y − ŷ)², penalizes large errors more. - Classification: zero–one error counts wrong classes; other losses can evaluate predicted probabilities.
A lower error is better for a loss. Accuracy is a score where higher is better, so do not compare “error” and “accuracy” without checking the direction and definition.
The generalization gap
generalization gap ≈ held-out error − training errorA small gap is not sufficient by itself: both errors could be high because the model underfits, the features are weak, or the data is noisy. A large gap suggests that the model fits the training sample better than it transfers to unseen examples, often because of overfitting or a flawed split.
Worked calculation
Suppose two models predict the same regression target. The residuals are:
| Model | Training residuals | Test residuals |
|---|---|---|
| A | 1, −2, 1, −2 | 2, −3 |
| B | 0, 1, 0, −1 | 4, −5 |
Using MSE:
- Model A training MSE =
(1² + 2² + 1² + 2²)/4 = 2.5. - Model A test MSE =
(2² + 3²)/2 = 6.5. - Model B training MSE =
(0² + 1² + 0² + 1²)/4 = 0.5. - Model B test MSE =
(4² + 5²)/2 = 20.5.
Model B fits the training sample better, but Model A has the lower estimated generalization error. The goal is not the smallest training error in isolation; it is reliable performance on the intended future data.
Training, validation, and test error
- Training error: used to understand fit and optimize parameters.
- Validation error: used during development to compare choices or tune hyperparameters.
- Test error: reported after choices are complete as an estimate of future performance.
If the test set is used repeatedly to choose models, it becomes part of the development feedback. Its reported error then becomes optimistic and no longer represents an untouched final estimate.2
What can make the estimate misleading?
A held-out score can still be misleading when:
- the test set is too small or unrepresentative;
- examples are dependent but split as if independent;
- time changes the data distribution;
- preprocessing or labels leak across the split; or
- the metric does not match the real cost.
Generalization is always relative to a population and a task. A model can generalize from one hospital to similar patients but fail after a sensor or policy change.
Exercise
Three models have these errors:
| Model | Training error | Validation error |
|---|---|---|
| A | 0.02 | 0.03 |
| B | 0.01 | 0.40 |
| C | 0.30 | 0.32 |
Which model is the most suspicious for overfitting? Which is the most suspicious for underfitting? Is A automatically the best model?
Answer
B is suspicious for overfitting: very low training error but much higher validation error. C is suspicious for underfitting: both errors are high and close. A is the best candidate among these numbers, but not automatically: check the split, metric, uncertainty, leakage, and performance on the intended population.
Exam lens
Definition pair: training error measures fit on observed training examples; generalization error is expected loss on new examples from the target distribution.
Diagram to reproduce: show training error generally decreasing as capacity increases, while validation/test error can decrease and then rise. Label the gap and the minimum validation region.
Common traps:
- zero training error does not imply zero generalization error;
- low training and validation error are different from low training error alone;
- test error is an estimate, not a mathematical guarantee;
- “error” must name its loss and data split.
Rapid revision
- Can I write the training-error formula?
- Can I explain why generalization error is an expectation we estimate?
- Can I calculate MSE from residuals?
- Can I explain how repeated test-set tuning invalidates the final estimate?
Sources
Footnotes
-
Mitchell, T. M. (1997). Machine Learning. McGraw-Hill — training performance, generalization, and learning problem formulation. ↩
-
scikit-learn, “Cross-validation: evaluating estimator performance.” https://scikit-learn.org/stable/modules/cross_validation.html — held-out evaluation and model-selection workflow. ↩ ↩2