Overfitting and Underfitting

1.7 — Overfitting and Underfitting

Recall first. A model gets 100% on training data and 60% on validation data. Is it good, bad, or impossible to tell? Name the missing evidence.

The central goal: fit the future

A model should learn structure that transfers to new examples. It should not merely memorize the training set, and it should not be so restricted that it misses the main pattern.

• Underfitting: the model is too simple, too constrained, or insufficiently trained to capture the useful structure. Training performance is poor, and validation/test performance is also poor.
• Good fit: training and validation/test performance are both acceptable, with a manageable gap.
• Overfitting: the model adapts too closely to the training sample, including noise or accidental details. Training performance is strong, but validation/test performance is substantially worse.¹

A score such as 100% training accuracy is not a diagnosis by itself. We need a validation or test comparison and a meaningful split.

Model complexity and the U-shaped validation curve

A typical pattern is:

error
  ^       validation/test error
  |          \      /
  |           \____/
  | training error \________
  +----------------------------> model complexity
       underfit   useful   overfit

As complexity increases, training error often decreases because the model has more freedom to fit the sample. Validation error may first decrease, reach a useful region, and then rise as the model begins to fit sample-specific noise. The exact curve is not guaranteed; it is a diagnostic pattern, not a law.

Recognizing the symptoms

A large train–validation gap is evidence to investigate, not proof of one cause. Leakage can produce a misleadingly good validation result rather than the ordinary overfitting pattern.

Causes of underfitting

• model capacity is too low;
• important features are missing or poorly represented;
• excessive regularization restricts the model;
• training stopped too early;
• the chosen objective cannot represent the task;
• the data contains strong noise or inconsistent labels.

Possible responses include improving features, choosing a more expressive model, reducing excessive regularization, or training appropriately. More complexity is not automatically the answer.

Causes of overfitting

• model capacity is too high relative to data;
• the training set is small or unrepresentative;
• the model is trained too long without validation monitoring;
• irrelevant features or noise are available;
• the practitioner tunes repeatedly against the same validation data;
• the data split is flawed or the target distribution changes.

Possible responses include collecting representative data, simplifying the model, regularization, early stopping, feature selection, data augmentation where valid, and cross-validation. The remedy must match the cause.

Learning curves

A learning curve plots performance against training-set size or training progress. It can reveal different situations:

• If training and validation errors are both high and close, adding more training time may not solve underfitting; richer features or a more suitable model may be needed.
• If training error is low but validation error is high, more representative data may reduce variance, although it is not guaranteed.
• If validation loss rises while training loss continues to fall, the model may be overfitting after that point.¹²

Do not confuse a learning curve with a single final score. The shape explains how performance changes.

Worked example — polynomial degree

Suppose we predict a curved physical measurement from one input using polynomial regression:

Degree 1 is likely underfitting: both errors are high. Degree 3 is the best candidate here: it reduces both errors. Degree 12 is likely overfitting: it nearly memorizes training observations but performs poorly on validation data.

The conclusion is not “always choose degree 3.” The useful complexity depends on the data, loss, split, and deployment distribution.

Exercise

Diagnose each pattern and suggest one response:

1. A shallow decision tree has high training and test error.
2. A very deep tree has zero training error and poor validation error.
3. A model has excellent validation accuracy, but the validation records contain future information.

Exam lens

Three-pattern answer:

• underfit = high train error + high validation error;
• good fit = both acceptable and reasonably close;
• overfit = low train error + much higher validation error.

Long-answer skeleton: define the two failures, draw error against complexity, give causes, then give matched remedies. State that validation/test performance—not training performance alone—decides generalization.

Common traps:

• overfitting is not simply “a large model”; it is poor transfer to new data;
• underfitting is not fixed by blindly collecting more epochs;
• regularization can reduce overfitting but can also cause underfitting if excessive;
• validation data can itself become overused.

Rapid revision

• Can I diagnose underfit and overfit from two error values?
• Can I sketch the complexity/error relationship?
• Can I name two causes and two remedies for each failure?
• Can I distinguish overfitting from data leakage?

Sources