Optimization: Gradients, the Hessian and Gradient Descent

A smooth function can only peak or bottom out where it stops rising and falling, so the first move in any optimization is to set the derivative to zero. Points where $f'(x) = 0$ are critical points. The derivative tells you they are flat; a second step tells you what kind of flat.

Suppose a desk can sell $q$ units at price $100 - 2q$, and producing them costs $20q + 50$. Profit is

Then $\pi'(q) = 80 - 4q$, which is zero at $q = 20$, and $\pi(20) = 750$.

The first derivative test checks the sign of $\pi'$ on either side: positive for $q < 20$, negative for $q > 20$, so profit climbs and then falls. That is a maximum. The second derivative test is faster: $\pi''(q) = -4 < 0$, so the curve bends down and the flat spot is a peak. In general $f'' > 0$ at a critical point means a local minimum and $f'' < 0$ means a local maximum.

When $f'' = 0$ the test says nothing. Both $x^3$ and $x^4$ have $f'(0) = f''(0) = 0$, yet $x^4$ has a minimum at 0 while $x^3$ keeps rising through it. A point where $f''$ changes sign, as $x^3$ does at 0, is an inflection point.

Two more checks catch most interview slips. A local extremum need not be the global one, and on a closed interval the endpoints are candidates too. The global answer is the best value among the critical points and the endpoints.