Bias, Variance and Regularization

You have 100 months of returns and 1000 candidate predictors. OLS solves the normal equations $X^TX\hat\beta = X^Ty$, and $X^TX$ is a $1000 \times 1000$ matrix of rank at most 100. It has no inverse. Worse, with more unknowns than equations there are infinitely many exact solutions: you can fit every observation perfectly, $R^2 = 1$, on pure noise.

The trouble starts well before $p > n$. Suppose the true signal is zero, the noise has variance $\sigma^2$, and you fit OLS with $p$ parameters on $n$ points. The fitted values are $Hy$, where the hat matrix $H$ has trace $p$. Then

With $n = 100$ and $p = 50$, the training error is $0.5\sigma^2$ and the error on new data at the same inputs is $1.5\sigma^2$. In sample the model looks twice as good as the truth; out of sample it is 50% worse than predicting zero. Each extra parameter bends the fit toward the particular noise it happened to see, and that noise doesn't repeat.

The rest of this lesson accepts a little bias in exchange for much less variance, so that a fit holds up on data it hasn't seen.