Pop Quiz. Let’s say you take a Covid-19 antibody test. The test has an accuracy of 90%. To be more specific, both its sensitivity (the proportion of actual positives that are correctly identified as such) and specificity (the proportion of actual negatives that are correctly identified as such) are 90%. The test comes back positive. What is the chance that you actually have antibodies to Covid?
a) 90%
b) 80%
c) The answer cannot be determined from the information given
The correct answer is C. You are missing one key piece of information: the rate of positive results in the entire population. Without this data, you cannot know the probability that you actually have Covid antibodies.
That’s right, you have to know how many people test positive in the population as a whole before you can judge the predictive value of a test. It’s called the base rate fallacy and it’s counter-intuitive, to say the least.
One way of understanding it is that the accuracy measurements of a test (sensitivity and specificity) answer one set of questions, and the Pop Quiz asks a different question. The accuracy measurements answer these questions:
(a) If I truly do have Covid antibodies, will I get a positive test result? (sensitivity)
(b) If I truly don’t have Covid antibodies, will I get a negative test result? (specificity)
But the Pop Quiz question turns it around:
(c) If I get a positive test result, do I truly have Covid antibodies? (In other words, how predictive is the test?)
[Along with the complimentary question:
(d) If I get a negative test result, do I truly not have Covid antibodies?]
And in order to answer (c), you need more information: the rate of antibodies in the general population (the base rate). To see why this is the case, let’s do a couple of examples.
Scenario 1. In a population of 10,000 people, let’s say 4% actually have Covid antibodies. That means we have 400 people with antibodies and 9600 without. If we test all 10,000 people:
- Of the 9600 without antibodies, 10% will come back with false positive results (since the test is 90% accurate). That results in 960 positive results (false positives) and 8640 negative results (true negatives).
- Of the 400 with antibodies, 10% will come back with false negative results (again, due to the accuracy of the test). That results in 40 negative results (false negatives) and 360 positive results (true positives).
Of the 10,000 people, 1320 (960+360) had positive results. But of these, only 360 actually had the antibodies; 960 were false positives (test error). So the odds that a positive test is correct is 360 / 1320, or 27.3%.
Scenario 2. Same thing, except now let’s say 20% of the population actually has antibodies. That means we have 2000 people with antibodies and 8000 without. (The test accuracy is the same as before, 90% sensitive and 90% specific.) If we test all 10,000 people:
- In the 8000 without antibodies, we get 800 positive results (false positives) and 7200 negative results (true negatives).
- In the 2000 with antibodies, we get 200 negative results (false negatives) and 1800 positive results (true positives).
Of the 10,000 people, 2600 (800+1800) had positive results. Of these, 1800 actually had the antibodies; 800 were false positives (test error). So the odds of the positive test being correct are now 1800 / 2600, or 69.2%.
That’s why you need to know the prevalence in the general population (base rate) before you can answer the quiz: the answer changes depending on this value. This dynamic comes up all the time, in cancer screening, drug testing, and so on. One rule is that the as the prevalence is more rare, predictive value of the test drops precipitously. We saw this a little with the examples above, where the odds changed from 69% to 27% as the prevalence dropped. If what you’re searching for is extremely rare, then even an extremely accurate test will result in far more false positives than true positives.
Example: you’ve been tested for a rare disease, using a test that is 99% accurate. But the disease only happens once in 10,000 people. If you get a positive result on the test, what are the chances you actually have the disease? Less than 1%! The other 99% of the time, it’s just a false positive.
On the other end of it, when the prevalence reaches 50% of the population, then the predictive value of the test equals its accuracy. As prevalence goes beyond 50%, the predictive value exceeds its accuracy.
All evidence is that Coronavirus exposure in the US is well under 50%. As long as it is, keep this in mind when you read about the accuracy of various tests. Even a test with high accuracy may not be predictive, due to low prevalence.
Great write up, really important to know these days. Just want to note that a test like this might be affected by the date of the test relative to when the infection began; you might test negative for antibodies in the first few weeks since exposure, because antibodies take time to develop. Or, after a few months, the antibodies might be present, but in very low numbers so the test reads negative. So even a perfect test will have limitations.