Forecasting Unlikely Events
At "A Dash" we have tried to develop two themes that are important both to traders and to individual investors:
- Experts add value in forecasting. The notion that everyone's opinion is equal, while popular on the web, can be costly to one's portfolio.
- Forecasting should be judged with respect to the difficulty of the task. Some problems have a high error rate because of their inherent characteristics. Improving the forecast is still valuable.
Getting a Fresh Perspective
Sometimes the best way to understand a problem is to step away from the specific issue and look at a comparable situation where everyone does not already have an opinion. When that exercise is finished, we can all consider whether it is a true analogy and what lessons might apply.
If readers will join in this exercise with an open mind, we expect some interesting results and discussion.
A Hypothetical Problem
Here is a problem that few have considered. Let us suppose that we have assembled a panel of experts on baseball. Let us also suppose that we have assembled a panel of baseball fans from a particular city. The team in that city is playing a home game tomorrow. We ask all of them the question:
Will the home team lose by more than three runs?
The Statistical History
Since baseball is such a wonderful game for statistics, we have some good information about this question. The chance of a home team loss by more than three runs is about 20%. (That is also the chance of a home victory by that margin. The home field advantage is offset by the fact that the home team does not bat in the bottom of the ninth if already winning, so their scoring is reduced by 1/9).
Our panel of experts, if challenged to predict a three-run loss, would all vote in the negative. They know the data. They would be wrong 20% of the time and correct 80% of the time. No panelist would vote in favor of the proposition. If the panelist votes were publicized in the newspaper, such a prediction would look foolish.
How Did the Experts Do?
If we did a statistical analysis asking only about the success of spotting three-run losses, we would conclude that the panel was terrible. They never made such a prediction, missing every case, even though such losses happened 20% of the time.
(Of course they also did not predict a lot of "false positives", but that was not the question).
Posing the Question Differently
Let us next suppose that we asked the panel to give us a probability estimate that the home team would lose by three runs. The base estimate of our experts would be 20%, but there could now be some variation. If an average team (my White Sox, sad to say) was playing a strong team like Boston or New York, the chances would be higher. If the opponents had an ace pitcher on the mound and the ChiSox were using someone recently called up from Charlotte, the odds might get close to 50-50. Even in extreme cases, however, the odds of a three-run victory do not get above 50%.
Let us suppose that an expert put the chances of the big loss at 40%, but the home team actually won the game! Was the expert wrong? Not necessarily. We cannot tell from a single game. The odds might well have been 40%. It would take many games of similar circumstances for us to judge the accuracy of the prediction.
Briefly put, our experts would never predict a three-run loss in a specific game, although their probability estimates would reflect the specific circumstances.
The Fan Panel
It is easy to imagine that the range of fan predictions would be different from that of the experts. When fans get down on their team or a specific pitcher, many of them expect the worst. The result is that some fans might predict a three-run loss on a given day. They are going by "feel" and momentum, and other factors.
They also make many loose forecasts that defy measurement. The "team is not hitting" and "fielding is sloppy". Performance will be uneven -- maybe poor. The bullpen has been erratic. The team is hard to predict.
The result is that the Fan Panel predicts more three-run losses than do the experts. They are better than the experts at predicting the big losses! (They also have many more "false positives.")
Conclusion
The average statistical expectation in any baseball game is a small margin of victory. The modal (most common) difference is one run. Three-run losses come from "random shocks" which sportswriters explain in detail the next day.
Predicting a very unlikely result like a three-run loss, even under the worst circumstances, is a losing proposition. This means that no expert handicapper would make such a prediction. As a result, the entire expert panel appears inept at predicting big losses. The fan panel would always have a few random people who made winning choices, so they would seem better than the experts on this problem.
There are plenty of implications for this important principle, including many which are important for investors. Some astute readers will have already made key inferences. We shall elaborate in future articles.
