Logistic Regression

A credit desk wants the probability that a borrower defaults, given the debt-to-income ratio $x$ in percent. The outcome is $y \in \{0, 1\}$. The quickest thing to try is ordinary least squares on the 0/1 labels, the linear probability model. Suppose it returns $\hat p = -0.15 + 0.012x$. At $x = 10$ it predicts $-0.03$, a negative probability. At $x = 100$ it predicts $1.05$. A line has no reason to stay inside $[0, 1]$, and its errors can't be homoscedastic either: for a 0/1 outcome, $\text{Var}(y \mid x) = p(1-p)$, which changes with $x$.

The fix is to model a quantity that lives on the whole real line. The odds $p/(1-p)$ run from 0 to $\infty$, and the log-odds, or logit, runs from $-\infty$ to $\infty$. Logistic regression sets the logit equal to a linear function:

The right-hand side is the sigmoid $\sigma(z) = 1/(1 + e^{-z})$, an S-curve that flattens near 0 and 1 and is steepest at $p = 1/2$. Written as a model, $y \sim \text{Bernoulli}(p)$ with $\text{logit}(p) = x^\top \beta$. It is a generalized linear model with the logit as its link.

Swap the sigmoid for the standard normal CDF $\Phi$ and you get probit. After rescaling the two curves are nearly indistinguishable. Logit coefficients come out roughly 1.6 to 1.8 times the probit ones, because the logistic distribution has standard deviation $\pi/\sqrt{3} \approx 1.81$ where the normal has 1. Logit is the usual default because its coefficients have the odds reading in the next section.