Regression Inference: Standard Errors, t and F

A regression reports $\hat\beta_1 = 2.5$. Is the true slope different from zero, or would another sample have given 0.3? To answer, treat the estimate as what it is: a function of noisy data. Write the model as $y = X\beta + \varepsilon$ with $E[\varepsilon \mid X] = 0$ and $\text{Var}(\varepsilon \mid X) = \sigma^2 I$. Substituting into the OLS formula from Least Squares as a Projection,

so $\hat\beta$ is unbiased and

The standard error of coefficient $j$ is the square root of the $j$th diagonal entry, with $\sigma^2$ replaced by its estimate $s^2 = \text{RSS}/(n-p)$, where $p$ counts every coefficient including the intercept. Dividing by $n-p$ fixes the same bias as the $n-1$ in the sample variance: the fit used up $p$ degrees of freedom.

For a simple regression the diagonal entry has a closed form:

That gives three levers: less residual noise, more spread in $x$, more data. Because of the $\sqrt n$, halving the standard error costs four times the data.

Worked example: $n = 27$, residual standard error $s = 4$, and $\sum (x_i - \bar x)^2 = 16$. Then $\text{SE}(\hat\beta_1) = 4/\sqrt{16} = 1$.