The residual is a deviation score measure of prediction error in case of regression.
The difference between an observed target and a predicted target in a regression analysis is known as the residual and is a measure of model accuracy.
The error term is an unobserved variable as:
In a scatterplot the vertical distance between a dot and the regression line reflects the amount of prediction error (known as the “residual”).
<math>e = Y - \hat{Y}</math>
where in a regression
The ingredients of prediction error are actually:
Bias and variance together gives us prediction error.
This difference can be expressed in term of variance and bias:
<math>e^2 = var(model) + var(chance) + bias</math>
where:
As the flexibility (order in complexity) of f increases, its variance increases, and its bias decreases. So choosing the flexibility based on average test error amounts to a bias-variance trade-off.
See Statistics - Bias-variance trade-off (between overfitting and underfitting)