Principal Component Analysis

Stack $n$ observations of $p$ variables into a data matrix $X$ ($n \times p$) and subtract each column's mean. The sample covariance matrix is

Project every row onto a unit vector $\mathbf{w}$ and each observation becomes a single number. The column of those numbers, $X\mathbf{w}$, has variance

PCA asks which unit $\mathbf{w}$ makes this as large as possible. $\Sigma$ is symmetric, so the spectral theorem hands us an orthonormal eigenbasis $\mathbf{q}_1, \dots, \mathbf{q}_p$ with eigenvalues $\lambda_1 \ge \dots \ge \lambda_p \ge 0$. Write $\mathbf{w} = \sum c_i \mathbf{q}_i$ with $\sum c_i^2 = 1$. Then $\mathbf{w}^T\Sigma\mathbf{w} = \sum \lambda_i c_i^2$, a weighted average of the eigenvalues, which is largest when all the weight sits on $\lambda_1$. The best direction is the top eigenvector and the variance it captures is $\lambda_1$. The best direction orthogonal to that one is $\mathbf{q}_2$, and so on down the list. These eigenvectors are the principal components.

Example: two stocks with daily return covariance $\Sigma = \begin{pmatrix} 5 & 2 \\ 2 & 2 \end{pmatrix}$ in $\%^2$. Trace 7 and determinant 6 give eigenvalues 6 and 1. The first PC is $(2,1)/\sqrt{5}$ with variance 6. For comparison, the first stock alone, $\mathbf{w} = (1,0)$, gives 5, and the equal-weight direction $(1,1)/\sqrt{2}$ gives $(5+2+2+2)/2 = 5.5$. No unit direction beats 6, and that single line carries $6/7 \approx 86\%$ of the total variance.