The alarm fired. How worried should you be?
A signal that is right 95% of the time can still be wrong most of the times it fires, because the thing it looks for is rare. Bayes' rule weighs the true alarms against the false ones.
- Count true and false alarms from a base rate, a sensitivity and a false-alarm rate
- Compute P(event | alarm) and P(event | silence)
- Use the odds form to find the cleanest false-alarm rate a target posterior allows
- Explain base-rate neglect and why doubling the base rate does not double the posterior
The intuition
A risk model raises an alarm on 90% of the days a market dislocation happens, and on 5% of ordinary days. Dislocations happen on 1% of days. The alarm has just fired. Out of 10,000 days, 100 are dislocations and the model catches 90 of them; 9,900 are ordinary and the model wrongly flags 495. So 585 alarms, of which 90 are real: a 15% chance today is a genuine dislocation, not 90%.
The intuitive answer, the sensitivity, answers the wrong question. Sensitivity is 'given a dislocation, does the alarm fire?' The question asked was 'given the alarm, is it a dislocation?' The two are linked by how rare the event is, and with ordinary days outnumbering dislocation days 99 to 1, even a small false-alarm rate produces more false alarms than true ones. Every 'the test came back positive' question is this calculation.
P(alarm) = base × sens + (1 − base) × fpr. P(event | alarm) = base × sens ÷ P(alarm). P(event | silence) = base × (1 − sens) ÷ (1 − P(alarm)). In odds: posterior odds = prior odds × (sens ÷ fpr). The false-alarm rate a target posterior q allows: fpr = base × sens × (1 − q) ÷ (q × (1 − base)).
Why it works
- The conventions here: a binary alarm that fires on a stated share of event days (sensitivity) and a stated share of ordinary days (the false-alarm rate). The base rate is the unconditional frequency of the event. Days are independent.
- Count in two buckets. True alarms come from event days: base × sens. False alarms come from ordinary days: (1 − base) × fpr. The posterior is true ÷ (true + false). Working in days out of 10,000 keeps the arithmetic honest.
- Silence is informative too. P(event | no alarm) = missed events ÷ silent days. A high sensitivity makes a quiet day genuinely reassuring even when the alarms themselves are mostly noise.
- The odds form separates the model from the world. Posterior odds = prior odds × likelihood ratio, and the likelihood ratio sens ÷ fpr belongs to the model alone. A rare event needs a very large likelihood ratio before an alarm is more likely right than wrong.
- Run it backwards for a specification. Set the posterior to the level the desk will act on, and the odds form gives the maximum false-alarm rate the model can have. Usually it is far below what people expect.
- Doubling the base rate doubles the odds, not the probability. A small posterior roughly doubles; a large one cannot, because probabilities are capped at one.
| True alarms: 1% × 90% | 0.90% of days |
| False alarms: 99% × 5% | 4.95% of days |
| P(alarm): 0.90% + 4.95% | 5.85% of days |
| P(dislocation | alarm): 0.90 ÷ 5.85 | 15.38% |
| P(dislocation | silence): 1% × 10% ÷ 94.15% | 0.106% |
| Odds form: prior odds 1/99 × LR 18 | 0.1818 to 1, i.e. 15.38% |
For an alarm to be right half the time, the false-alarm rate would have to be 1% × 90% × 50% ÷ (50% × 99%) = 0.91%. Doubling the base rate to 2% takes the posterior to 26.87%, not 30.77%.
The formulas
Alarms come from two places: event days and ordinary days.
True alarms as a share of all alarms.
Missed events as a share of silent days.
The likelihood ratio belongs to the model; the prior odds to the world.
The odds form solved for the false-alarm rate.
Worked example
Count the true alarms and the false alarms as shares of all days, then divide. The follow-up asks why the answer is so far below the sensitivity.
See it move
Same model. Change how rare the event is, how often the model catches it, and how often it cries wolf.
- Raise the base rate. True alarms rise, false alarms fall, and both posteriors rise: an alarm means more, and silence means less.
- Raise the sensitivity. True alarms rise, P(dislocation | alarm) rises a little, and P(dislocation | silence) falls: sensitivity mostly buys you trustworthy silence.
- Raise the false-alarm rate. False alarms rise and P(dislocation | alarm) falls, while P(dislocation | silence) rises, because the silent days now contain a larger share of the missed events.
Run it backwards
The desk will only act if an alarm is right a stated share of the time. What is the highest false-alarm rate the model can have?
Turn the target posterior into odds, divide by the prior odds to get the likelihood ratio required, and the false-alarm rate is sensitivity divided by that ratio.
The follow-up checks the model's actual false-alarm rate against the ceiling, and says what to do if it fails.
Traps
Say it in the interview
“The model is 90% accurate on dislocation days and it just fired. Is today a dislocation?”
Check yourself
5 fresh questions, with new numbers. Answer each one correctly to finish the chapter. Get one wrong and you will see the full working, then you can try it again with new numbers.
Answers within 1% are marked right. Type the number; $, %, x and M are fine. First tries count toward Learned: the topic is Learned once every chapter is done and 75% of first tries were right.
- Count two buckets: base × sens true alarms, (1 − base) × fpr false ones.
- P(event | alarm) = true ÷ (true + false); P(event | silence) = missed ÷ silent.
- Posterior odds = prior odds × sens ÷ fpr. The likelihood ratio is the model; the prior is the world.
- Sensitivity is not the answer. The base rate is the number to ask for.