Statistics & Regression

Lesson 6 of 9

Multicollinearity and Interactions

What a regression coefficient means when predictors are correlated, how the variance inflation factor measures the damage, and how interaction terms let one variable's slope depend on another.

What a multiple regression coefficient means

In a multiple regression y=β0+β1x1+β2x2+εy = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \varepsilon, the coefficient β1\beta_1 is a partial coefficient: the change in E[y]E[y] when x1x_1 rises by one unit and x2x_2 stays fixed. It is generally different from the slope you get by regressing yy on x1x_1 alone. The two agree when x1x_1 and x2x_2 are uncorrelated, or when the partial coefficient on x2x_2 happens to be zero, as in the example below.

The Frisch-Waugh-Lovell theorem makes "holding fixed" concrete. Regress x1x_1 on x2x_2 and keep the residual x~1\tilde x_1, the part of x1x_1 that x2x_2 cannot explain. Then β^1\hat\beta_1 equals the slope of yy on x~1\tilde x_1. The regression learns about x1x_1 only from the variation it does not share with x2x_2.

With standardized variables there is a closed form. Let r1yr_{1y} and r2yr_{2y} be each predictor's correlation with yy, and ρ\rho the correlation between the predictors:

β^1=r1y−ρ r2y1−ρ2,β^2=r2y−ρ r1y1−ρ2\hat\beta_1 = \frac{r_{1y} - \rho\, r_{2y}}{1-\rho^2}, \qquad \hat\beta_2 = \frac{r_{2y} - \rho\, r_{1y}}{1-\rho^2}

Worked example: a 12-month and a 6-month momentum signal have ρ=0.9\rho = 0.9. Their correlations with the target are r1y=0.5r_{1y} = 0.5 and r2y=0.45r_{2y} = 0.45. Then β^1=(0.5−0.405)/0.19=0.5\hat\beta_1 = (0.5 - 0.405)/0.19 = 0.5 and β^2=(0.45−0.45)/0.19=0\hat\beta_2 = (0.45 - 0.45)/0.19 = 0. The 6-month signal looks good on its own, yet once you know the 12-month signal it adds nothing: its entire correlation with the target is accounted for by 0.9×0.5=0.450.9 \times 0.5 = 0.45. So the answer to "which one matters?" is the predictor whose unshared part still predicts yy.