The Perils of Data-Fitting

We are delighted to see the Felix Salmon piece (highlighted by Abnormal Returns) reviewing the supposed relationship between Japan in the 90's and the current US market.  Check out the article to see the "great fit."

Here at "A Dash" we have frequently exposed such  bogus relationships, even when the loud voices took the other side.  There is a simple process for generating a misleading relationship:

1. Examine a bunch of charts.  Look for variables that are all correlated to the economy in some way.
2. Find a couple of prominent features to the charts.  It is especially good if one of them (the leader) shows a move not yet reflected in the other.
3. Adjust the scales to make the prominent features line up.  Ignore any other activity.
4. Disseminate the predicted next move as widely as possible.

We exposed this process in the "Batman" chart allegedly showing a relationship between bank lending criteria and personal consumption expenditures.  This widely publicized chart was never defended by any of the original big-time proponents.  The collapse in PCE did not occur in the predicted time frame, as we showed here.  No one was keeping score.

It was an obvious case of data-fitting, although our analysis did not get the buzz that we see for Felix Salmon's piece.  Perhaps it will do better in the book version.

What You Do Not Understand

Let us assume that everyone now sees the peril of twisting old chart data to fit a pattern.  This is a lesson that can be applied to something you do not already know.

When someone reaches into history for an analogy -- -it is the same thing.  It is back-fitting of data, using the human mind, the most powerful computer.

When someone presents a trading system -- based upon many runs of old data, it is back-fitting.  The spectacular returns are not real.

Our Take

It is a shame that people are drawn to bad charts, bad analogies, and bad systems.

Too bad for them.  For the rest of us, it is a time of opportunity.

Comments

[Patrick]

I find it hard to believe that the people making these charts do not realize the con they are trying to pull. So, why make them, and disseminate them, when they know that other knowledgeable people will think them stupid? To make money, like everything. I can think of three ways:

1. To sell systems to the gullible (but this will have the data behind a pay-for wall and is easily seen).
2. To sell books or generate web traffic. Dogs of the Dow would be an example of the former, whereas the Foolish Four would be an example of the latter. There are blackjack books that give out important information that is worth the cost of the book. Probably the same is true of investment books and sites. But there is also a ton of garbage out there.
3. To get people to trade with you. You put out false or misleading data hoping to get retail investors to trade one way, with you trading the other. Only if you expect to make a lot of money would you take the reputation hit. Possibly, if it works, your reputation with your insiders will grow even if your reputation with others falls. An example of this would be the fact you only hear gold investing commercials on Bloomberg at gold market tops.

[RB]

If only people just stuck to looking at Bangladesh's butter production.