Problem bank

Problem 237 of 333MediumStatisticsP237

Ridge and lasso with orthonormal predictors

  1. Suppose the predictors are orthonormal, XTX=IX^TX = I, and let b=XTyb = X^Ty be the OLS coefficients.

    (a) Show that ridge, min⁡β∥y−Xβ∥2+λ∥β∥2\min_\beta \|y - X\beta\|^2 + \lambda\|\beta\|^2, gives β^j=bj/(1+λ)\hat\beta_j = b_j/(1 + \lambda).

    (b) Show that lasso, min⁡β∥y−Xβ∥2+λ∥β∥1\min_\beta \|y - X\beta\|^2 + \lambda\|\beta\|_1, gives β^j=sign(bj)max⁡(∣bj∣−λ/2,0)\hat\beta_j = \text{sign}(b_j)\max(|b_j| - \lambda/2, 0).

    (c) With b=(2.4,−0.6,1.1,−3.0)b = (2.4, -0.6, 1.1, -3.0) and λ=2\lambda = 2, compute both estimates.