Sep 26, 2020

What is a false positive? Heck, what’s a statistical test?

Here I explain testing, the R number and give you some tips on how to read statistics. Currently, there are many graphs and a lot of numbers flying around but how can we be sure they are any good?

In statistical testing, we start with some sort of hypothesis of what might happen and then we test this to see if we are right or if there is a change that proves us wrong.

In statistical hypothesis testing, we first decide on a hypothesis. We split up a hypothesis to enable us to prove it true or false.

We use Ho to say nothing has changed and H1 to say something has changed.

When we perform a test we use a lot of parameters to bound our test but there is always the chance of an error. Causes of errors can include such things as; the data can be slightly odd due to many factors, we could set our boundaries such that we incorrectly classified a result or we did the wrong test on the data we had. There are many ways to generate errors.

To describe the errors we might see, we use Type I and Type II errors.

A Type I error is the rejection of H0 if it is true. i.e an innocent person being convicted

A Type II error is the rejection of H1. i.e a guilty person is not convicted.

Why is all this important in clinical testing.

Testing is not perfect and new clinical tests can be fraught with errors when released into the real world. It is not uncommon to see fairly high levels of false positives or false negatives (or Type I and Type II Errors). Anyone who has used a pregnancy test and got a faulty result can attest to this.

This is why, when using statistics and doing tests, and then presenting the results we have to not only use highly qualified, independent people who have integrity but also we must have laws to protect test subjects and ensure we collect data in a very systematic way.

The R number

Let’s discuss the R number very quickly. This measure comes from some complex maths and with all complex maths, it has a lot of assumptions behind it that means we should not rely on it solely. The R number has been given as a description of how many cases there are i.e how fast can a disease reproduce and spread (Interesting fact – this wasn’t the original purpose of this measure). Now, this number tells us how fast a disease reproduces and even spreads, and in the case of a cold or flu it will be high, is this a problem – no because the illness spreads fast but lethality is low. Thereby we get immunity and the virus goes away. If we had, say, ebola, a high r number, and a high lethality mean many many deaths as the disease spreads quickly and kills at a high rate.

As with all measures, there are problems with R. The R number can be an average of an entire population and not take into account local R numbers. There are times we would be very interested in an R number and times when we are not.  Concentrating on one measure alone can obscure other very important factors such as other illnesses or diseases spiking, trends in hospitals the removal of healthcare due to a misunderstood R measure. When producing indicator measures and KPI’s for a business we never rely on one measure, we use several and we use them, within context, to create a bigger picture. This forms a more robust analysis and shows us where we can concentrate on interventions.

Ways to test statistics

Given that statistics can really be bent and shaped any which way to produce a nice chart, if you are presented with statistics it is important to look at the following:

• Who produced them
• Why were they produced?
• What motivation is behind their production
• Do the graphs have titles and axis labels
• Is there an explanation or context around these numbers/graphs
• Does the argument look like a reflection of the real world
• Is there a paper/explanation in plain English I can read
• Is this unnecessarily complex and if so what might be being hidden (Tip, most statistics that are required for analysis aren’t that complicated so if it is complicated, why is it)
• What references are quoted on the research and do they seem reasonable (Could they be bias or all supporting one argument)
• Tip: A paper should really discuss both sides of the argument before forming a conclusion
• If there is a particular indicator being concentrated on why might this be
• Are other measures presented
• Is the bigger picture described

I hope that implementing these tools and techniques allows you to pull back the curtain on the analysis we see each day.