Are You Smarter than a Doctor?

If so, you are also smarter than a major fund manager.

Once again investors are confronted with a LIST OF WARNINGS!!!

The basic idea is that the end of the world is near.  The evidence is a laundry list of events occuring in close proximity with former occasions of the end of the world.

The reasoning offered is seductively persuasive.  The background story is great -- sort of like the Hindenburg Omen or the Death Cross.

Your BS detector should be on red alert when you see this stuff, but can you really see the error?  It seems so persuasive.....

The Doctors' Problem

As I have often suggested, investors benefit from stepping outside of their normal world.  Try to think clearly about the following problem:

1% of women at age forty who participate in routine screening have breast cancer.  80% of women with breast cancer will get positive mammographies.  9.6% of women without breast cancer will also get positive mammographies.  A woman in this age group had a positive mammography in a routine screening.  What is the probability that she actually has breast cancer?

Only 15% of doctors answered the question accurately.

Give it a try yourself.  If you grasp the concept, the answer will be easy.

If you can solve this problem, you will have a great feel for the error in the list of warnings.  Please do not post exact answers in the comments, except to say that you have solved it.  Email the answers to jmiller at newarc dot com and I'll hand out the awards!  Some of my readers will nail this one.

I know that readers like things wrapped into a a nice one-article package, but I am not going to do that today.  There are several important lessons here and you won't get them if I put it in a single piece:

1. Hardly anyone can do probability problems accurately.  You need to see how seductively wrong this is.
2. Even the smartest market participants make many errors on causation and inference -- the includes some high-profile managers and high-priced analysts.
3. If you can figure out problems like this, you have a real advantage.

References

I created and used a lot of problems like this in my teaching, but I saw a really good source with a more modern example.  I'll catch up on giving credit with the solution.

No fair searching to help your answer!  Do your own work!

And take pride if you solve this problem.

 

 

Comments

[oldprof]

That's my 4sf total as well:)

Jeff

[Gary]

Good point. In my experience with economists in a variety of contexts - some of them love their models far more than they care about the reality the models are supposed to describe. Between that, and people dredging for data that supports their preconceived notions, it's hard for statistical work of value to be produced at all.

[Mike C]

Thanks Gary. I didn't immediately think about the difference in sample sizes across the two splits.

As much as we would like to reduce market/economic analyis to physics/engineering problems, I think there is an inherent complexity full of feedback loops that make drawing exact conclusions difficult. Soros reflexivity comes to mind.

Ultimately, with any sort of statistical analysis of the market/economy, I think of a few things.

1. It is better to be roughly right than precisely wrong. And errors should be tilted towards safeguarding capital. Louise Yamada says there are only two losses in investing, loss of capital, and loss of opportunity, and there are always more opportunities.

2. Does the relationship make sense? It is impossible to form a logical reason why the price of butter in Malaysia should impact the U.S. economy/market. On the other hand, things like credit spreads and ISM numbers make sense that they accurately point north on the compass.

[StillStudying]

As I was going to St. Ives, I met a man with 7 wives....

[Dave]

Meanwhile the (me) less confident demand some feedback :-)

In lieu of a direct answer, when expressed with 4sf, the sum of the (my!) digits is 24.

Anyone agree?

[Gary]

I don't know anything about Kobe or Lebron, but imagine the following sample data:

Lebron shoots 466/555 for the first half (84%) while Kobe misses a bunch of games for injury and shoots 142/167 (85%). Kobe has the higher average.

Second half of the season, Lebron 329/417 (79%), but Kobe comes back & shoots 424/533 (80%). Kobe has the higher average.

Overall, Kobe shoots 566/700 for 81%, while Lebron shoots 792/972 for 82% - Lebron has the higher average for the season.

This effect can happen in any partitioned group - not just "first half" vs. "second half" of a sports season - but treatment groups in medical trials, groups you're analyzing for investment etc. So a medical treatment that looks more effective than an alternate for group and and for group B may look less effective than the alternate for the groups together.

In fact, distressingly enough, mathematically ANY group can be partitioned in a way that produces this effect. Meaning that if you look hard enough, you can actually invert the inferences you draw from your data!

The more different the sizes of the cells in the partitioned data (in this case, halves of the season and also different players), the stronger the effect can be. You can even have double paradoxes, I believe, where each player also shot better in the first half than the second half of the season, but overall it was better in the second half (not the case in this instance).

[oldprof]

Francisco -- It's OK. Several people missed the request. I need to figure out a better method:)

Meanwhile, I am still getting answers from people who get the blog via email.

Jeff

[Mike C]

This screening difficulty, for instance, or Simpson's Paradox-style difficulties, where Kobe can have a higher free-throw percentage than Lebron for each half of the season, but Lebron will have a higher percentage over the whole season.

Gary, can you show the math behind this one?

[Francisco]

I missed the part saying not to publish the answer. I apologize for doing so.

[Gary]

The statistical problems of comparing groups of different sizes are manifold:

This screening difficulty, for instance, or Simpson's Paradox-style difficulties, where Kobe can have a higher free-throw percentage than Lebron for each half of the season, but Lebron will have a higher percentage over the whole season.

Other size-driven paradoxes?

[Kevin]

Sent my answer in and apologize for being late to the party. I work as an engineer and we're in the midst of product testing so time is limited.

Hope to see Dr. Miller's answer soon. Thanks for a great financial site!

[Dave]

Answer in the post!
As put in the email, my personal take on this is...

Here's hoping that having had my two kids do the (UK) A-level maths I've been reminded of enough stats to get this right, especially as I sometimes remind them of the statement by Google's Val Harian "As Data is next big cheap thing, being statistically quallified is best way to go!" [or something like that].

Cheers
Dave (UK)

[oldprof]

JC -- I have temporarily "unpublished" a couple of comments describing an answer so that more people can enjoy the problem.

I'll "republish" them in a day or two. Meanwhile, as you note, it would be nice if everyone would just send solutions via email for later recognition.

Thanks,

Jeff

[JC Morris]

Two thoughts.

First, Yes, I solved the riddle and it is as SI posted.

Second, (@SI), what part of "Please do not post exact answers in the comments, except to say that you have solved it" do you not understand? Our good and generous host was, I think, trying to see how many readers would leap to a quick and incorrect conclusion (and be willing to post it). But now, unless they agree with you and I, they likely won't post. C'mon, man. Of course people have known how to do it for as long as there has been arithmetic. That's not the point. The point is that even the simplest statistics are often misunderstood by intelligent (though perhaps innumerate) individuals. And the world of finance is full of both statistics and innumerate people.

[Gary]

I solved it, but i've seen the problem before.

[rs]

"All your ba_es are belong to me."
Laplace

[Francisco]

Sorry for typing the same text twice. I would remove one but I am unable to edit the text.