Hypothesis Tests, p-values and Confidence Intervals

A trader tells you 116 of her last 200 trades made money. Is she good, or could a coin have done that? Hypothesis testing makes the question precise. The null hypothesis $H_0$ is the boring explanation: each trade wins with probability $p = 0.5$. The alternative $H_1$ is her claim, $p > 0.5$.

Next you need a test statistic, a number that measures how far the data sit from what $H_0$ predicts. Under $H_0$ the win count has mean $100$ and standard deviation $\sqrt{200 \cdot 0.25} \approx 7.07$, so

The p-value is the probability, computed assuming $H_0$ is true, of a statistic at least as extreme as the one observed. Here $P(Z \ge 2.26) \approx 0.012$ (the exact binomial tail is 0.014). If you fixed a significance level $\alpha = 0.05$ in advance, then $0.012 < 0.05$ and you reject $H_0$.

This is a one-tailed test, because only a high win rate supports her claim. If you were asking whether a coin is biased in either direction, you would count both tails: the two-tailed p-value is $2 \times 0.012 \approx 0.024$. Pick the tail before looking at the data. Choosing it afterwards quietly halves your p-value.

The interview trap is the interpretation. A p-value of 0.012 does not mean there is a 1.2% chance she has no skill. It is $P(\text{data this extreme} \mid H_0)$. Turning that into $P(H_0 \mid \text{data})$ needs a prior on how common skilled traders are, which is the reversal covered in Bayes' Theorem.