Causal Inference and False Positives

Probability and causal inference are two of the most difficult subjects for most students of research methods.  The investor needs to understand a few basic principles to avoid being bamboozled by false claims.  My mission this week has been to provide help with some of the basic skills.

I am going to start with the solution to the "doctor problem."  If you have not already tried to solve this problem, you will enjoy the article more if you test your skill first from the original article.

I'll review the problem, highlight some reader good readers solutions, and finally draw some investment conclusions.

The Problem

Here was the question, answered correctly by only 15% of doctors:

1% of women at age forty who participate in routine screening have breast cancer.  80% of women with breast cancer will get positive mammographies.  9.6% of women without breast cancer will also get positive mammographies.  A woman in this age group had a positive mammography in a routine screening.  What is the probability that she actually has breast cancer?

 

The Solution and Reader Results

The correct answer is 7.764%.  Readers of "A Dash" did pretty well with 36% emailing me with correct answers (allowing for some rounding).  20% believed the probability was 90.4%.  25% thought the chances were 1 percent or less.  14% of readers thought the odds were about 10%.

Putting this another way, even our astute readership was mostly wrong and had a wide variation of responses.  My point was to show how difficult it is to do this kind of inference.  I appreciate everyone who sent in an answer.  Most people do not even attempt such a question.

I borrowed the problem from this helpful essay on Bayesian inference.  There is a careful explanation of the solution and a method that you might use.  Some of our readers (including Nemo Publicus, who blogs at https://self-evident.org/) used an explicit and rigorous Bayesian approach.

Let A = "she has breast cancer"

Let B = "she tests positive"

We seek P(A|B).

Basic Bayesian law says:

P(A) * P(B|A) = P(B) * P(A|B)

(because both sides equal P(A AND B), which is the only way I can ever remember it)

We are given:

P(A) = 0.01

P(B|A) = 0.8

P(B|~A) = 0.096

We need P(B), which we can get from:

P(B) = P(B|A)*P(A) + P(B|~A)*P(~A)

      = 0.8*0.01 + 0.096*0.99

      = .008 + .09504

      = .10304

That is, the probability of a randomly chosen woman testing positive is just north of 10%, which is only slightly higher than the 9.6% false positive rate.  Not surprising since 99/100 women do not have cancer.

Substitute into P(A|B) = P(A) * P(B|A) / P(B)

= 0.01 * 0.8 / .10304

= .07764

So she has around a 7.8% chance of having cancer.

Reader CG put it this way:

Assumptions:

1% Women @ 40 Have Breast Cancer / 99% Women @ 40 Don't Have Breast Cancer

80% of Women w/ Breast Cancer Test Positive

9.6% of Women w/o Breast Cancer Test Positive 

 Question: 

What is the probability that she has breast cancer?

 Solution: 

(Women @ 40 w/ Breast Cancer that Test Positive) / (Women @ 40 that Test Positive)

(Women @ 40 w/ Breast Cancer that Test Positive) / ((Women @ 40 w/o Breast Cancer that Test Positive) + (Women @ 40 w/ Breast Cancer that Test Positive))

W = Women @ 40

(0.01W*0.8)/((0.99W*0.096)+(0.01W*0.8)) = 7.76%

 Others constructed a table of an example population.  Here is the submission of F.G., a CFA.

 

And finally, here is a very succinct and accurate solution from reader GK.

80% of the 1% with breast cancer is .8%
9.6% of the 99% without breast cancer is 9.504%
9.504% and .8% is 10.304%
.8% of 10.34% is 7.7639%

In other words, a positive mamogram means a woman has a 92.2361% chance that she does not have breast cancer.

One More Example

Now that you have solved that one, you are well prepared to think about this situation.  Suppose that 1/3 of all US residents over age 12 have tried pot (ignoring whether or not they inhaled).   Further suppose that 1% of this same population are heroin addicts.  Finally, those who never try pot also do not become heroin addicts.

The statement that all heroin addicts have used pot is accurate, but may be misleading.  The reason is that the overwhelming majority of pot smokers (over 95%) do not become heroin addicts.  Despite this, it is a common argument from the "just say no" crowd.

A useful causal statement must begin with the suggested cause and look at all of the resulting outcomes, with odds.  And that is just the starting point.

An Application for Investors

Let us try the statement that "there has never been a recession without falling stock prices."  Or "every recession has been preceded by a yield curve slope of less than X."  These may be true statements, but they are just as misleading as the assertion about heroin.  Contrast this sort of contention with the analysis of Eddy Elfenbein in his fine weekly market review: 

Remember that the stock market isn’t a great predictor of the economy. Since 1945, the S&P 500 has fallen 17% or more 14 times, and nine of those times have seen recessions. The bond market, however, has a better track record. Bloomberg writes, “the economy has never contracted with the difference between 10-year and 30-year Treasury yields as wide as the current 1.38 percentage points, or 138 basis points.” That’s pretty eye-opening.

Eddy, as usual, is careful and accurate in formulating the question.

It is also not helpful to pile on even more "indicators" using this same format.  The causal inference is still backwards, and the additional factors, often tweaked to specific values, are based on looking at all of the data.  That's cheating!

You can never prove such pundits wrong with existing facts, since the overfitted back-testing has exhausted all of the data.

Special Warning!!

Beware when the number of "indicators" gets large, especially when compared to the number of events to be explained.  There are not many recessions, so using a lot of variables is a methodological blunder related to degrees of freedom -- the subject for another day.  A method that provides insight with one or two variables is actually much more persuasive than an omen or syndrome or laundry list that uses many factors.

This is exactly the opposite of what intuition would suggest, which is why it is so important to know.

Examples

There are many such examples in the everyday discussion of investments.  Some of these are bullish arguments and some are bearish, although most of the current crop seem to be about recessions.  For those seeking an example, check out last year's article where I tried to explain this by showing the flaws in the Hindenburg Omen.

I hope this week's series will help equip readers in their reading and decision-making.  Once again, I appreciate the participation in the problem, and the many kind words from readers.  I especially enjoyed comments from those who admitted that they had not tried a problem like this in years.  If I do this again, I need a better method for accepting entries, getting permission to acknowledge responses, and recognizing the best answers.

 

Comments

[oldprof]

StillStudying -- I think the results from our readers were quite good.

I was trying to illustrate probability, but the application in screening is important, as you note. The emphasis is on finding real positives. The false positives can then get further testing.

In investing, a false positive about a recession may lead to a poor decision, just as much as failing to miss the signs.

You are correct to note that the standard varies.

Thanks,

Jeff

[oldprof]

rs-- I had a nice chat with Mish at the Kauffman Foundation meeting this year. He is passionate and committed, and he has a successful business model for his blog.

And yes, he is one of those who are certain about recession chances.

Jeff

[oldprof]

bfuruta -- I am delighted at your skepticism about the odds.

Concerning the liquidity trap debate, I am a regular reader of Krugman and DeLong, so I am familiar with their argument. Both are serious scholars of top rank. I happen not to find them as useful for investors.

My objection mostly comes from the fact that people who do not really understand economics (and I am not putting you in that group) and do not want to consider data, simply grab a term like this and use it to bring the analysis to a close.

Thanks again for commenting so carefully, and I hope this clarifies my obscure point.

Jeff

[oldprof]

HG -- Point taken! I agree about implying false precision. Mostly I was trying to show people how to do this sort of calculation.

I actually treated every answer of 8% or so as "correct" in my scoring.

Thanks for reminding us about one of the most important elements in quantitative analysis.

Jeff

[HG]

I have a bone to pick with your answer. The answer, in my opinion is 7.8%. We don't have enough significant digits to make the answer more precise than that. My students nowadays seem to have never been even introduced to the concept of significant digits, so it's a pet peeve of mine. Sorry for the rant, but I think understanding precision is an important part of quantitative analysis.

[rs]

RB
Thanks for the Q. The question has even have appeared on a 6th grade exam here believe it or not. Though it's a different type of puzzle, it was still fun; you can tell the probabilistic muscles have been stretched by the newfound hesitation in the face of seemingly innocent numbers.

[cheap nfl jerseys]

The correct answer is 7.764%. Readers of "A Dash" did pretty well with 36% emailing me with correct answers (allowing for some rounding). 20% believed the probability was 90.4%. 25% thought the chances were 1 percent or less. 14% of readers thought the odds were about 10%.

[RB]

I saw this problem only now. It reminded me of the other time I recently exercised my Bayesian skills recently - to solve the problem described on page 21 here . The other gem in this survey was of course,
A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
which only 12% answered correctly.

[StillStudying]

Jeff: From your survey, 66% of us (your readers) pegged the solution at 10% or less. Good enough for government work, especially if some just eyeballed the situation rather than going to find a calculator and working it out. Pretty darn good. I'd say for investing purposes a metric with a 10% chance of predicting a possible outcome vs one with a 7.8% chance gets the same (failing) grade.

Which goes a long way towards explaining why routine mammograms might not be the holy grail....

[oldprof]

A valued friend (who also submitted a correct answer to the problem) has the following suggestion for future questions and entries:

"I see the problem. If you write to each person for permission, it may take a long time until you can write the next post. Even if you don't really need permission, you don't want to annoy the customer base or start a flame war or anything like that.

How about this: Make the default be opting out. When making the initial invitation to submit answers, say nothing will be posted unless the writer includes something like:

OK to use my answers but no name in blog post.
OK to use my answers and real name in blog post.
OK to use my answers with name BigFly in blog post.

You could include examples in your text, formatted for easy copy and paste as a template. After a few times, a short note and a link to a page with explanation and example template should do the trick.

When you write up the next post, say there were other really nice examples, but the writer did not opt in."

My friend has explained this well. It would be fun to do some problems and recognize the most astute readers.

Maybe I can get my friend to write a comment himself!

Jeff

[rs]

Here's one who "knows" we are already in a recession (note the tone and certainty in the article coupled with source for each point for convenience):

http://globaleconomicanalysis.blogspot.com/2011/09/global-recession-right-here-right-now.html

These indications are the priors like the 1% of women with cancer, but what about the conditional probabilities that change the likelihood?

It's been really interesting to see how the excessive optimism in May has eased off till the point of recession calls now.

[bfuruta]

Jeff, in reply to your questions:

1) Yes, my BS meter went off with the wording used in the piece: specifically "slam dunk" and "long history." I also wondered when reading Eddy's piece, without taking the time to look it up, how many times the spread between the 30-year and the 10-year Treasuries was 138 basis points or more. The reliability of any indicator is low when the number of cases is small.

2) No, of course I don’t think there is a 100% chance of a recession; just as Eddy said there is a chance of falling back into recession, even as he cited the Bloomberg piece. Eddy is indeed citing facts. My point is that he is picking the facts that support his opinion and ignoring those that do not.

I am not trained in economics, so I try to learn from blogs like yours and others. Paul Krugman has been writing about the liquidity trap, with data and a model. The predictions from the model have been accurate so far. From your comment, I have to ask you a question. Do you think "liquidity trap" is an empty catch phrase with no data behind it?

Jeff, I want to thank you for your work. As I have said before, I appreciate your approach and the information you share in this blog. I will certainly look for your weekend article.

[oldprof]

bfuruta --

Eddy is citing a fact from the article, and he did it accurately. He did not try to portray the source as having a certain opinion.

I have a couple of questions for you:

1) How many recessions have there been since 1948? Doesn't your BS meter go off when someone cites the "long history" of the indicator? If not, I have failed in this series of articles!

2) Do you think that we now have a 100% chance of a recession in the next year? That this will always happen when GDP goes below 1.5%?

I'll put aside the policy questions for the moment (although look for this weekend's article) because this is about data analysis. I think that people use arguments like "liquidity trap" because they do not want to use data. Most people are more comfortable following the parrots than doing actual analysis. People are easily fooled.

BTW -- Your source cited "Bloomberg" not a person. I am sure you know that Bloomberg quotes a wide variety of sources of varying reliability. They leave it up to you to figure it out. I am curious about the person behind the quote.

Thanks for joining in, and for participating in the problem. As you say, time will tell.

Jeff

[bfuruta]

Jeff, you wrote, "Eddy, as usual, is careful and accurate in formulating the question."

I could make a case that Eddy has confirmation bias. The Bloomberg paragraph he quoted appeared under the heading "Growth Slowed" and had two parts. Here is the entire paragraph:

While Commerce Department figures showed gross domestic product climbed at a less-than-previously estimated 1 percent annual rate from April through June, the economy has never contracted with the difference between 10-year and 30-year Treasury yields as wide as the current 1.38 percentage points, or 138 basis points.

It is a balanced paragraph. Eddy left out the part about low GDP. He also didn't bother looking for any implications of GDP slowing that much. Here is what he could have found:

http://www.fxstreet.com/news/forex-news/article.aspx?storyid=30f27c15-c71b-4d60-916e-1330fd22c308

Source: Bloomberg

The YoY change in real GDP, presently at 1.5%, is a slam dunk indicator of recession: "Since 1948, every time the four-quarter change has fallen below 2 percent, the economy has entered a recession. It's hard to argue against an indicator with such a long history of accuracy" Bloomberg reports. Adding that "every time real YoY GDP has dropped below 1.5%, this has led to a negative nonfarm payroll number."

I personally think that you and Eddy are not giving enough weight to the larger economic environment. We are in a liquidity trap. As Eddy does state, it is fear that is driving down the yield on the 10-year Treasury. Too much debt and lack of aggregate demand are the problems, not high interest rates due to the Fed fighting inflation. The FOMC can do very little to help the situation. The politicians here and in Europe, unfortunately, are not up to the task. Unlike Eddy, I think there is good reason for the fear.

Time will tell.