accuracy | A Place of Sand

Now that Covid-19 rapid home tests are widely available, it is important to consider how to interpret their results. In particular, I’m going to address two common misconceptions.

To keep things grounded, let’s use some actual data. We’ll assume a false positive rate of 1% and a false negative rate of 35%. These numbers are consistent with a March, 2021 metastudy [1]. We’ll denote the false positive rate $E_p=0.01$ and the false negative rate $E_n=0.35$ .

It may be tempting to assume from these numbers that a positive rapid covid test result means you’re 99% likely to be infected, and a negative result means you’re 65% likely not to be. Neither need be the case. In particular,

A positive result does increase the probability you have Covid, but by how much depends on your previous prior. This in turn depends on how you are using the test. Are you just randomly testing yourself, or do you have some strong reason to believe you may be infected?
A negative result has little practical impact on the probability you have Covid.

These may seem counterintuitive or downright contradictory. Nonetheless, both are true. They follow from Bayes’ thm.

Note that when I say that the test increases or decreases “the probability you have Covid,” I refer to knowledge not fact. You either have or do not have Covid, and taking the test obviously does not change this fact. The test simply changes your knowledge of it.

Also note that the limitations on inference I will describe do not detract from the general utility of such tests. Used correctly, they can be extremely valuable. Moreover, from a behavioral standpoint, even a modest non-infinitesimal probability of being infected may be enough to motivate medical review, further testing, or self-quarantine.

Let’s denote by $C$ the event of having Covid, and by $T$ the event of testing positive for it. $P(C)$ is the prior probability of having covid. It is your pre-test estimate based on everything you know. For convenience, we’ll often use $\mu$ to denote $P(C)$ .

If you have no information and are conducting a random test, then it may be reasonable to use the general local infection rate as $P(C)$ . If you have reason to believe yourself infected, a higher rate (such as the fraction of symptomatic people who test positive in your area) may be more suitable. $P(C)$ should reflect the best information you have prior to taking the test.

The test adds information to your prior $P(C)$ , updating it to a posterior probability of infection $P(C|O)$ , where $O$ denotes the outcome of the test: either $T$ or $\neg T$ .

In our notation, $P(\neg T|C)= E_n$ and $P(T|\neg C)= E_p$ . These numbers are properties of the test, independent of the individuals being tested. For example, the manufacturer could test 1000 swabs known to be infected with covid from a petri dish, and $E_n$ would be the number which tested negative divided by 1000. Similarly, they could test 1000 clean swabs, and $E_p$ would be the number which tested positive divided by 1000.

What we care about are the posterior probabilities: (1) the probability $P(C|T)$ that you are infected given that you tested positive, and (2) the probability that you are not infected given that you tested negative $P(\neg C|\neg T)$ . I.e. the probabilities that the test correctly reflects your infection status.

Bayes’ Thm tells us that $P(A|B)= \frac{P(B|A)P(A)}{P(B)}$ , a direct consequence of the fact that $P(A|B)P(B)= P(B|A)P(A)= P(A\cap B)$ .

If you test positive, what is the probability you have Covid? $P(C|T)= \frac{P(T|C)P(C)}{P(T|C)P(C)+P(T|\neg C)P(\neg C)}$ , which is $\frac{(1-E_n)\mu}{(1-E_n)\mu+E_p(1-\mu)}$ . The prior of infection was $\mu$ , so you have improved your knowledge by a factor of $\frac{(1-E_n)}{(1-E_n)\mu+E_p(1-\mu)}$ . For $\mu$ small relative to $E_p$ , this is approximately $\frac{E_p}{1-E_n}$ .

Suppose you randomly tested yourself in MA. According to data from Johns Hopkins [2], at the time of this writing there have been around 48,000 new cases reported in MA over the last 28 days. MA has a population of around 7,000,000. It is reasonable to assume that the actual case rate is twice that reported (in the early days of Covid, the unreported factor was much higher, but let’s assume it presently is only $1\times$ ).

Le’ts also assume that any given case tests positive for 14 days. I.e., 24,000 of those cases would test positive at any given time in the 4 week period (of course, not all fit neatly into the 28 day window, but if we assume similar rates before and after, this approach is fine). Including the unreported cases, we then have 48,000 active cases at any given time. We thus have a state-wide infection rate of $\frac{48000}{7000000}\approx 0.00685$ , or about 0.7%. We will define $\mu_{MA}\equiv 0.00685$ .

Using this prior, a positive test means you are $45\times$ more likely to be infected post-test than pre-test. This seems significant! Unfortunately, the actual probability is $P(C|T)= 0.31$ .

This may seem terribly counterintuitive. After all, the test had a 1% false positive rate. Shouldn’t you be 99% certain you have Covid if you test positive? Well, suppose a million people take the test. With a 0.00685 unconditional probability of infection, we expect 6850 of those people to be infected. $E_n=0.35$ , so only 4453 of those will test positive.

However, even with a tiny false positive rate of $E_p=0.01$ , 9932 people who are not infected also will test positive. The problem is that there are so many more uninfected people being tested that $E_p=0.01$ still generates lots of false positives. If you test positive, you could be in the 9932 people or the 4453 people. Your probability of being infected is $\frac{4453}{9932+4453}= 0.31$ .

Returning to the general case, suppose you test negative. What is the probability you do not have Covid? $P(\neg C|\neg T)= \frac{P(\neg T|\neg C)P(\neg C)}{P(\neg T|\neg C)P(\neg C)+P(\neg T|C)P(C)}= \frac{(1-E_p)(1-\mu)}{(1-E_p)(1-\mu)+E_n\mu}$ . For small $\mu$ this is approximately $1$ unless $E_p$ is very close to $1$ . Specifically, it expands to $1-\frac{E_n}{(1-E_p)}\mu+O(\mu^2)$ .

Under $\mu_{MA}$ as the prior, the probability of being uninfected post-test is 0.99757 vs 0.9932 pre-test. For all practical purposes, our knowledge has not improved.

This too may seem counterintuitive. As an analogy, suppose in some fictional land earthquakes are very rare. Half of them are preceded by a strong tremor the day before (and such a tremor always heralds a coming earthquake), but the other half are unheralded.

If you feel a strong tremor, then you know with certainty than an earthquake is coming the next day. Suppose you don’t feel a strong tremor. Does that mean you should be more confident that an earthquake won’t hit the next day? Not really. Your chance of an earthquake has not decreased by a factor of two. Earthquakes were very rare to begin with, so the default prediction that there wouldn’t be one only is marginally changed by the absence of a tremor the day before.

Of course, $\mu_{MA}$ generally is not the correct prior to use. If you take the test randomly or for no particular reason, then your local version of $\mu_{MA}$ may be suitable. However, if you have a reason to take the test then your $\mu$ is likely to be much higher.

Graphs 1 and 2 below illustrate the information introduced by a positive or negative test result as a function of the choice of prior. In each, the difference in probability is the distance between the posterior and prior graphs. The prior obviously is a straight line since we are plotting it against itself (as the $x$ -axis). Note that graph 1 has an abbreviated $x$ -axis because $P(C|T)$ plateaus quickly.

From graph 1, it is clear that except for small priors (such as the general infection rate in an area with very low incidence), a positive result adds a lot of information. For $\mu>0.05$ , it provides near certainty of infection.

From graph 2, we see that a negative result never adds terribly much information. When the prior is 1 or 0, we already know the answer, and the Bayesian update does nothing. The largest gain is a little over 0.2, but that’s only attained when the prior is quite high. In fact, there’s not much improvement at all until the prior is over 0.1. If you’re 10% sure you already have covid, a home test will help but you probably should see a doctor anyway.

Note that these considerations are less applicable to PCR tests, which can have sufficiently small $E_p$ and $E_n$ to result in near-perfect information for any realistic prior.

One last point should be addressed. How can tests with manufacturer-specific false positive and false negative rates depend on your initial guess at your infection probability? If you pick an unconditional local infection rate as your prior, how could they depend on the choice of locale (such as MA in our example)? That seems to make no sense. What if we use a smaller locale or a bigger one?

The answer is that the outcome of the test does not depend on such things. It is a chemical test being performed on a particular sample from a particular person. Like any other experiment, it yields a piece of data. The difference arises in what use we make of that data. Bayesian probability tells us how to incorporate the information into our previous knowledge, converting a prior to a posterior. This depends on that knowledge — i.e. the prior. How we interpret the result depends on our assumptions.

A couple of caveats to our analysis:

The irrelevance of a negative result only applies when you have no prior information other than some (low) general infection rate. If you do have symptoms or have recently been exposed or have any other reason to employ a higher prior probability of infection, then a negative result can convey significantly more information. Our dismissal of its worth was contingent on a very low prior.
Even in the presence of a very low prior probability of infection, general testing of students or other individuals is not without value. Our discussion applies only to the interpretation of an individual test result. In aggregate, the use of such tests still would produce a reasonable amount of information. Even if only a few positive cases are caught as a result and the overall exposure rate is lowered only a little, the effect can be substantial. Pathogen propagation is a highly nonlinear process, and a small change in one of the parameters can have a very large effect. One caution, however. If the results aren’t understood for what they are, overconfidence can result. The aggregate use of testing can have a substantial negative effect if it results in relaxation of other precautions due to overconfidence resulting from a misunderstanding of the information content of those test results.