Orthogonality

The projection of $\mathbf{b}$ onto the line through $\mathbf{a}$ is the closest point on that line:

The error $\mathbf{b} - \text{proj}_{\mathbf{a}}(\mathbf{b})$ is orthogonal to $\mathbf{a}$; "closest" and "perpendicular error" are the same condition. Projecting onto a subspace (like the column space of a matrix) works identically: drop a perpendicular.