You’re being tricked – by your own medical mind. Pretest probabilities mean more to your clinical work than you might think. While sensitivity, specificity and P value, the ‘gold standard’ of statistical validity, is not as universally applicable as many of us, including many scientist, assume.

This is an attempt of explaing pretest probabilities, the basis of Bayesian statistics and why it is important to our everyday clinical work. Don’t fall asleep yet, but read through and see what you think.

**Starting point.**

This article in Nature: Scientific method: Statistical errors discusses the p-value and why it is often used incorrectly. The article is well worth a read as it explains in simple words why the pre-test probability of a hypothesis being true or false will often mean far more than the p-value of the end result. What they’re touching on is Bayesian statistics vs Frequentist statistics. Saywhat?!?

**Frequentists vs Bayesians**

Frequentist statistics is what we usually refer to as statistics: pure numbers. “Objective” numbers referring to a normal frequency distribution – symbolised by the Bell curve. But to apply it correctly in real life settings, you often need to adjust your numbers.

Enter Bayesian statistics. It deals with conditioned risk – how the risk is conditioned by pre-test probabilities and facts that will affect the outcome and skew it compared to pure frequentist statistics.

Pretty much a fancy way of working out what we try to do every day in medicine: Adjust general, categorical statistics to your specific patient. Sort of like “subjective” numbers. It will skew your Bell curve. And it will help you navigate in the world of statistics by adding relevant information that gives new perspective to your statistics. A conditioned probability is simply a probability based on some extra background information.

**Bayes and common sense**

Bayesian statistics has a wide application in medicine – and maybe we should use it even more. Not that it’s a new idea, Thomas Bayes worked on his theories in the 18th century – but it seems computers and new thinking helped bring his theories back into practice.

The most basic concept is the one where you tune your statistics to your patient. Risk of ischemic heart disease has a certain prevalence in the general population, but for a particular patient diabetes, gender, smoking etc skews that probability. And when a patient comes in with general symptoms of illness, the normally rare diagnosis of malaria springs to mind if the patient has been traveling in malaria endemic areas. Both are basic examples of pre-test probabilities ‘conditioning’ your statistics.

Often, we don’t need to apply the whole Bayesian formal framework, but just the Bayesian way of thinking. Most of the time, that just means common sense. In a way, Bayesian statistics is the rules and math of common sense. But more powerful than that, Bayesian statistics sometimes end up giving surprising results where common sense can lead you astray.

**Back to the p. Remember pre-test conditions**

One area where common sense might trick you, is tests with low pre-test probabilities, like the Nature article talks about. It is important to keep in mind that a very unlikely hypothesis will need more than a positive test result and a nice, low p to prove them. Consider this experiment by the excellent XKCD. You need to summon your nerd powers to get this:

OK, you didn’t get it. Luckily, the XKCD cartoon is explained by real nerds here. The essence is that the chance of the sun exploding is extremely low compared to the chance that the test is false positive (the dice throw: 1/36th of the time).

**Bayes in everyday medicine**

For a similar problem in the medical world, consider a setting similar to what we encounter in clinical practice every day:

Suppose a drug test is 99% sensitive and 99% specific. That is, the test will produce 99% true positive results for drug users and 99% true negative results for non-drug users. Suppose that 0.5% of people are users of the drug. If a randomly selected individual tests positive, what is the probability he or she is a user?

Normally, we would trust that test. The patient tested positive, so he took the drug. Or did he? Bayesian statistics can help tune the answer. The real, Bayesian, probability is – surprisingly – only 33%.

It follows the same logic as the XKCD cartoon. Because drug use occurs less frequent than the test being false positive, your end result is fraught with risk of error. There is a 1% chance of a false positive test result, but only a 0.5% chance the person being tested is a drug user. So the false positive will come out twice as often as you will encounter an actual drug user. Wikipedia explains it in detail here. Bayes’ thinking would dictate that you account for this pre-test flaw in your assessment of the test result.

**Ace of Bayes**

Still, we tend to get tricked to believe the answer when the sensitivity/specificity is looking good, or if the p’s are all nice and low. Bayes helps correcting this and skew the result by taking into account relevant pre-test conditions – hopefully skewing your answer towards something more meaningful.

In the settings mentioned above, Bayesian statistics can help put a number on our clinical/common sense judgement, to weight different diagnosis against each other. I find the concept of Bayesian statistics fascinating, and a useful way of thinking to try to navigate through tests, articles and statistics trying to apply it all to my patients and clinical practice.

*For an easy-to-understand introduction to Bayesian statistics, have a look at this online book: Think Bayes. It’s not written for doctors, but the first few chapters, before it goes deeper into programming, is an excellent description of Bayes theorem and the logic behind it and how to apply it.*

*Also, to see a great example of how Bayesian statistics can come up with surprising, but correct, answers – answers that common sense just won’t understand – read the Monty Hall problem. The Monty Hall problem will drive you mad.*

Love it. I had to google the ‘all your base are belong to us’ quip (wasnt familiar with that meme)…..and the XKCD cartoon is super geeky

…yet this explanation is one of the shortest and most clear that I have seen and should be required reading for any clinician.

Great writing, good topic…very useful. Keep it up!

Thanks Tim! The post is a bit different from our more usual critical care stuff, but came as a result of me struggling with statistics and Bayes. And XKCD helped me… I find writing a post is the best way to get my head around things. If I was able to somewhat explain the thinking behind Bayes in simple terms, I reached my goal.

We often hide some more or less obscure references in our posts, so we’re always happy when someone gets them. And XKCD is so over the top geeky that just the geekiness in itself makes me smile sometimes. I’ll be very disappointed if the creator of XKCD doesn’t look like an older version of the stereotyped geeks from an 80’s high school comedy.

Anyway, the thinking behind Bayes statistics is proving more and more useful the more I apply it. It helps build some common sense into statistics and relating it to my everyday clinical work.

Dr. David Newman’s The NNT site makes this very concrete and applicable. Check out the diagnosis reviews.

http://www.thennt.com/

Thanks. The NNT is a favourite of ScanCrit. A typical article review wouldn’t necessarily touch upon the problems mentioned in this Bayes post, but the smart minds at NNT often incorporates this kind of thinking into their reviews of papers.

First, best title EVAH for a blog post.

So much for SPPIN (SPecific Positive, rule IN) and SNNOUT (SeNsitive Negative, rule OUT). In the instructive example above shows *some* value of a positive test, by improving the odds by 66x, at the expense of FN>TP. While a negative test improves the already-unlikely odds of excluding the disease, the clinical scenario would have to justify (risk of missing?) getting a test adds next to nothing to playing the odds.

It was instructive, if depressing, to create a simple spreadsheet to simulate various scenarios based on pre-test probability.

I recently read Daniel Kahneman’s “Thinking, Fast and Slow”. Among the central points is the ginormous extent to which human intuition is absolutely horrible when it comes to statistics. Our deliberative side can compensate, but will take the “lazy” approach when possible. Hence, the p-value “Seal of Approach” produces confidence even when the stated conclusion is XKCD-worthy.

FWIW, for the last 2 years Stanford University has offered a free on-line course in statistics in early June. Based on what I saw, it looked worthwhile. I had to bow out to study for a recert exam. Hopefully they will offer it again next year, as I am now officially a “dropout”.

Thx to Allie G for the NNT site; it was also well worth the visit.

Thanks for your comments, interesting thoughts on ordering tests vs playing the odds. The Kahneman book is genius, making us more conscious of our auto/”lazy” side.

And I will certainly look into the Stanford online statistics course. Thanks for the tip!

Pingback: The LITFL Review 149

Fasciniating. Love the Monty Hall problem. But, in clinical practice I have follwoing comments:

1) How often do you know the exact sensitivity and specificity of a test?

2) More importantly, how precisely do you know the exact prevelance of a condition?

Say for example using HbA1c to diagnose Diabetes. You could know a reasonable amount about the sensitivity and specificity (there are lab variations) but how do you know prevalence of diabetes in society? Chlamydia? Hep B?? All unknowns or guesstimates at best

[Late reply:] I agree with that assessment. The lab guys can sometimes help with a tests sens/spec. And you can often find statistics for the prevalence in the general population (which might or might not be the same as the probability for this patient).

But in general, for me, it’s more a state of mind when ordering/reviewing tests: Is the random lab test really anything to worry about? If everything else looks fine, is the CRP anything to worry about (it’s not even specific)? Is it worthwhile to do screening tests for a rare disease in a patient with low probability for that disease – just to be sure?

Have a good reason for ordering tests, and be sceptical when reviewing them. Re-order or seek further evidence if tests don’t match the patient. I think we usually do this by common sense – Bayes is just putting it into a system, and proving common sense has some merit.

Often we trust numbers as facts – this is a reminder that they’re not.