Underfitting happens when a model does not capture enough of the useful structure in the data. Overfitting happens when a model fits the training data too specifically and fails to generalize as well to unseen observations.
Think Beyond Training Performance
The goal of most predictive machine learning projects is not simply to achieve excellent performance on the observations used to train the model.
The more important question is whether the model has learned patterns that remain useful when it encounters new data from the relevant population or process.
Underfitting
The model is too limited, poorly specified, or insufficiently trained to capture important patterns in the data.
Overfitting
The model fits the training data very closely but captures noise or training-specific details that do not generalize well.
Good training performance is useful, but it should be considered alongside performance on data that was not used to fit the model.
What Is Underfitting?
Underfitting occurs when a model fails to learn enough of the relationship between the available features and the target.
As a result, the model may perform poorly even on the training data.
A Relationship That Is Too Simple
Imagine that the relationship between an input variable and an outcome contains a strong curved pattern.
If the selected model can represent only a very simple relationship, it may miss important structure and produce substantial errors on both training and unseen data.
Underfitting may occur when:
- the model is too simple for the underlying pattern;
- important predictive features are missing;
- regularization is too strong;
- the chosen representation of the data is inadequate; or
- some iterative models have not been trained sufficiently.
What Is Overfitting?
Overfitting occurs when a model adapts too closely to the particular training data rather than learning patterns that generalize well.
The model may capture random variation, unusual observations, or highly specific combinations of features.
A Very Deep Decision Tree
A decision tree can continue creating increasingly specific splits until small groups of training observations are separated.
This can produce extremely strong training performance while giving noticeably weaker performance on new observations.
A model that makes almost no errors on its training data may still perform poorly when presented with unseen data. Training performance should not be interpreted in isolation.
Underfitting vs Overfitting at a Glance
| Characteristic | Underfitting | Overfitting |
|---|---|---|
| Training performance | Often poor | Often very strong |
| Validation performance | Often poor | Noticeably worse than training performance |
| Model behavior | Misses useful patterns | Fits training-specific details too closely |
| Typical concern | Insufficient learning or flexibility | Poor generalization |
| Possible response | Increase useful model capacity or information | Control complexity and improve validation |
These are general patterns rather than rigid diagnostic rules. Model behavior depends on the task, metric, dataset, and evaluation procedure.
The Real Goal Is Generalization
Generalization refers to how well a model performs on relevant observations that were not used to fit it.
Suppose two models are evaluated on the same classification problem:
| Model | Training Accuracy | Validation Accuracy |
|---|---|---|
| Model A | 99% | 76% |
| Model B | 91% | 88% |
Model A has the higher training accuracy, but it also has a much larger gap between training and validation performance.
Model B shows more similar performance across the two sets in this example. That observation would be relevant when investigating generalization, although the final choice should consider suitable metrics, uncertainty, test performance, and the actual project objective.
How Can You Recognize Overfitting or Underfitting?
Comparing performance across training and validation data is one of the most useful starting points.
Possible Underfitting
Training performance is weak and validation performance is also weak.
Possible Overfitting
Training performance is substantially better than validation performance.
However, the size of an acceptable gap depends on the dataset, metric, noise level, model, and application. There is no single percentage difference that defines overfitting for every project.
A performance gap is evidence worth investigating, not a diagnosis that should be made without considering the full evaluation setup.
Why Does Overfitting Happen?
Overfitting can result from several interacting factors rather than one single mistake.
Model complexity
A highly flexible model may be able to represent useful patterns, but it may also fit noise when the available data does not adequately constrain that flexibility.
Limited training data
When the dataset is small relative to the complexity of the task, the model has fewer observations from which to learn stable patterns.
Too many weak or irrelevant features
Additional features are not automatically useful. Some may provide opportunities for the model to fit accidental relationships.
Noisy data
Measurement errors, inconsistent labels, or random variation can make it harder to distinguish stable signal from noise.
Repeated tuning against the same validation data
If many modeling decisions are made by repeatedly checking the same validation set, the development process can gradually adapt to that particular validation data.
How Can Overfitting Be Reduced?
The appropriate response depends on why the model is overfitting. Several strategies may help.
Control Model Complexity
Restrict unnecessary flexibility through model-specific settings such as tree depth or network architecture.
Use Regularization
Penalize or constrain model complexity where the selected method supports it.
Improve the Data
More representative observations or better-quality features can help the model learn more stable patterns.
Use Appropriate Validation
Cross-validation or another suitable validation strategy can provide a stronger basis for model selection.
Other approaches may include:
- removing genuinely unhelpful features;
- feature selection;
- data augmentation in suitable domains;
- early stopping for iterative models;
- pruning decision trees;
- dropout in appropriate neural-network settings; and
- using ensembles where appropriate.
Regularization, pruning, dropout, feature selection, and other techniques solve different problems. The response should match the model and the evidence you observe.
How Can Underfitting Be Addressed?
If a model cannot capture important patterns even in the training data, making it even more restrictive may make the problem worse.
Depending on the situation, you might consider:
- using a more flexible model;
- adding useful features;
- improving feature representations;
- reducing excessive regularization;
- adjusting restrictive model settings; or
- allowing sufficient training for models that learn iteratively.
Before increasing complexity, also check whether poor performance is actually caused by data quality problems, inappropriate features, an unsuitable metric, or an error in the modeling pipeline.
Learning Curves Can Reveal Useful Patterns
Learning curves show model performance as some aspect of training changes, such as the amount of training data or the number of training iterations.
One common learning-curve approach compares training and validation performance as the training set grows.
Large Persistent Performance Gap
Suppose training performance remains very strong while validation performance remains substantially weaker as more training observations are added.
That pattern may indicate a generalization problem worth investigating.
Both Perform Poorly
If training and validation performance are both poor and relatively close, the model may be unable to capture enough useful structure.
Learning curves should be interpreted alongside knowledge of the data, metric, model, and validation strategy rather than through a single universal rule.
Three Practical Examples
Decision Tree Classification
A tree achieves 100% training accuracy but substantially lower validation accuracy.
This large gap suggests that the tree may be fitting the training observations too specifically. Tree depth, minimum samples per leaf, pruning, data quality, and validation strategy are reasonable areas to investigate.
A Model That Is Too Restrictive
A model produces large errors on both its training and validation data, while exploratory analysis suggests meaningful structure remains uncaptured.
This may be consistent with underfitting. The features, model flexibility, preprocessing, and model assumptions should be reviewed.
Neural Network Training
During training, the training loss continues to decrease, while validation loss eventually begins increasing.
This can be a sign that additional training is improving the fit to the training data without improving generalization. Early stopping may be one technique worth considering.
Common Mistakes to Avoid
1. Choosing the model with the best training score
Training performance alone does not tell you how well the model will generalize.
2. Assuming a complex model is automatically better
More complexity can increase flexibility, but it can also make it easier to fit noise or unstable patterns.
3. Assuming a simpler model is automatically better
Simplicity can help generalization, but a model that is too restrictive may underfit.
4. Tuning repeatedly on the test set
Once test results influence repeated modeling decisions, the test data is no longer functioning as an untouched final evaluation.
5. Trying to fix overfitting without checking for leakage
Unrealistically strong performance can sometimes be caused by target leakage or incorrect data splitting rather than model complexity alone.
6. Looking only at accuracy
Accuracy may be misleading for some problems, particularly imbalanced classification. Use metrics that reflect the project objective and the consequences of different errors.
7. Assuming more data always solves overfitting
Additional representative data can help in many situations, but poor features, leakage, label problems, distribution shifts, or inappropriate model choices may still need to be addressed.
Final Model Generalization Checklist
The objective is not to maximize training performance at any cost. It is to build a model that captures useful patterns and performs reliably on relevant unseen data.
Having Trouble With Model Performance?
Share your dataset structure, model, training and validation results, evaluation metrics, and project requirements. The results can then be reviewed to understand whether overfitting, underfitting, data leakage, or another issue may be affecting the model.