Probability

Lesson 14 of 14

Generating Functions

Moment and probability generating functions: reading off moments, E[exp(X)] for a normal, adding independent variables by multiplying, dice totals as polynomials, and random sums.

One function that holds every moment

The moment generating function (MGF) of a random variable XX is

MX(t)=E[etX]M_X(t) = E[e^{tX}]

defined for the values of tt where the expectation is finite. Expand the exponential inside the expectation and take expectations term by term:

MX(t)=1+t E[X]+t22!E[X2]+t33!E[X3]+⋯M_X(t) = 1 + t\,E[X] + \frac{t^2}{2!}E[X^2] + \frac{t^3}{3!}E[X^3] + \cdots

So the nnth moment is the nnth derivative at zero, E[Xn]=MX(n)(0)E[X^n] = M_X^{(n)}(0). Equivalently, it is n!n! times the coefficient of tnt^n.

Take X∼Exponential(λ)X \sim \text{Exponential}(\lambda). Then

MX(t)=∫0∞etx λe−λx dx=λλ−t,t<λM_X(t) = \int_0^\infty e^{tx}\,\lambda e^{-\lambda x}\,dx = \frac{\lambda}{\lambda - t}, \quad t < \lambda

The integral blows up for t≥λt \ge \lambda. That is allowed: an MGF only needs to be finite on some interval around 0. Writing λλ−t=∑n≥0(t/λ)n\frac{\lambda}{\lambda - t} = \sum_{n \ge 0} (t/\lambda)^n and matching coefficients gives E[Xn]=n!/λnE[X^n] = n!/\lambda^n in one line. With λ=2\lambda = 2: E[X]=0.5E[X] = 0.5 and E[X2]=0.5E[X^2] = 0.5, so Var(X)=0.25\text{Var}(X) = 0.25, and E[X3]=6/8=0.75E[X^3] = 6/8 = 0.75. By integration by parts that would be three separate calculations.

For a discrete XX the integral becomes a sum. A Poisson(λ)\text{Poisson}(\lambda) variable gives

MX(t)=∑k≥0etk e−λλkk!=eλ(et−1)M_X(t) = \sum_{k \ge 0} e^{tk}\,\frac{e^{-\lambda}\lambda^k}{k!} = e^{\lambda(e^t - 1)}

and differentiating once at t=0t = 0 returns λ\lambda, as it should.