If a test to detect a disease whose prevalence is one in a thousand has a false positive rate of 5 percent, what is the chance that a person found to have a positive result actually has the disease, assuming you know nothing about the person’s symptoms or signs?
Almost half of the respondents (consisting of "20 house officers, 20 fourth-year medical students and 20 attending physicians, selected in 67 consecutive hallway encounters at four Harvard Medical School teaching hospitals") answered 95%. Only 11 participants got the correct answer: approximately 2%.
This isn't a trick question, but it is a tricky question because most people fail to take into account the prevelance of the disease (i.e., it afflicts, on average, one in every thousand people). In more technical language, we are dealing with conditional probabilities and not just marginal (i.e., non-conditional or simple) probabilities. The standard mathematical technique to deal with such problems is Bayes' Theorem (named after its discoverer, the Reverend Thomas Bayes). But that requires a whole lesson, or series of lessons, on its own.
Here's a simple, common-sense approach to the problem:
To begin with, assume that the test yields no false negatives. Suppose we test a randomly selected group of 1000 people. Based on the given information, we would expect just one of these people to have the disease and therefore give a true positive test. We would expect 5% of the remaining 999, or roughly 1000, healthy people to also test positive, i.e., about 50 false positive tests. In other words, out of the 51 positive tests, only one would be a true positive. Therefore the chance that a person found to have a positive result actually has the disease is 1/51, or approximately 2%.