Least Squares as a Projection

You observe $n$ pairs $(x_i, y_i)$ and model them as $y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$, where the $\varepsilon_i$ are noise. Stack the rows and it becomes

where $y$ is an $n$-vector, $X$ is the $n \times p$ design matrix (first column all ones for the intercept, one column per predictor) and $\beta$ holds the $p$ coefficients. Least squares picks the $\hat\beta$ that minimizes the sum of squared residuals $\|y - X\beta\|^2 = \sum_i (y_i - x_i^T \beta)^2$, where $x_i^T$ is row $i$ of $X$. Squares make the problem smooth with a closed-form answer, and when the noise is normal the same $\hat\beta$ comes out of maximum likelihood, as the Maximum Likelihood lesson shows.

Take five points: $x = (1,2,3,4,5)$, $y = (2,4,5,4,5)$. The design matrix has rows $(1, x_i)$. The useful move is to stop picturing five points in the plane and treat $y$ as a single vector in $\mathbb{R}^5$. Every choice of $\beta$ produces a vector $X\beta$, and the set of all of them is the column space of $X$, a 2-dimensional plane sitting inside 5-dimensional space. No line passes through all five points, so $y$ lies off that plane, and finding the best line becomes a geometry question: which point on the plane is closest to $y$?