Misusing the Most Powerful Computer

There is a perfect storm tempting many traders and investors.  The increased power of computers, the easy availability of data, and the user-friendliness of software have made it possible for nearly anyone to backtest trading systems.  The result is that people with no background or training in research methods are using the most powerful tools, but not knowing how to do it.

The sad result is the story of the system trader we covered in an earlier post, and the need for a scientific method for using the tools.  One of the comments to that post did a nice job of summarizing the best approach -- saving lots of out-of-sample data, making sure the method works in different eras and markets, testing equity curves during the relevant periods, etc.

Here we introduce a new idea, one that I have never seen before.  Perhaps readers will alert me to some other mention of this notion, so that I can cite it (since this is a blog about a book).

Suppose I told you that my computer discovered a certain technical "set-up".  Let us say that it was a double bottom, followed by a five-month rally in the stock or index.  The wise system tester might ask how many such instances there were, the comparative results, and expect an out-of-sample test after discovering an apparent relationship.  Such are the methods used by system gurus like James Altucher (whose book on trading systems we recommend in our reading list) and our own Vince Castelli.

Computer systems are evaluated with scientific skepticism and rigorous demands on testing -- and rightly so.  That is the world of the system trader.

Most market participants do not have the requisite skill set to evaluate systems.  They do, however, have another "skill" that is exceedingly dangerous -- looking at charts.  Everyone believes in his own ability to look at a chart, see trends, see breakouts, and see correlations.

They use the most powerful computer -- the human mind -- to follow a process vaguely similar to the development of a trading system.  The human analyst takes the current market situation and seeks out some past situation that seems similar.  Instead of using a computer technique, the human comes up with an old chart and compares it to the new one.

There are multiple problems:

• No one asks how many such "set-ups" there were, or what happened in all of the cases.
• There is no question about whether there are enough cases to form a conclusion.
• There is no out-of-sample testing.

The effect on the average reader, including market professionals, is very powerful.  The charts seem quite similar.  It is very much like the behavioral finance concept of anchoring, where a totally irrelevant fact predisposes humans to accept the fact as a base point for reality.  In the behavioral science literature a random number is often used as the base point.  Even though people know that it is random, it still has a powerful effect.

In the case of the old chart, the human parses through history to find the desired example.  Since almost no one understands how to test this, but all think they know how to read charts, the effect is powerful.  There is often little effort to compare the fundamental similarities and differences between the two time periods.

Sometimes the chartist offers several different stocks or indices from the same period.  Since the indices are all highly correlated, this actually provides no additional information, but it seems to make the argument more powerful.

While on vacation last week I read an analysis of this type at Barry Ritholtz's site, The Big Picture.  Barry, who is more skilled than most on behavioral finance traps, thinks that this is relevant to the current market.  In fact, Barry cites similar work in Barron's.

At "A Dash" we believe that such comparisons lack the requisite testing, the sort that we would routinely perform on technical predictions.  Barry is doing what everyone does, and he does it well.  Our objection is that the method is unsound.

Expert humans, even the most astute fund managers, are unduly influenced when someone does what we call human data dredging.

Since the influence of this is powerful, it provides an opportunity for those who reject the approach.  Let us be completely clear.  We are not saying that the conclusion is false.  Our position is that the analysis provides no useful additional information, yet it influences many in a specific direction.

One of the major themes at "A Dash" is that astute Wall Street professionals are subject to the same behavioral finance problems as individual investors.  Knowledge of the literature does not necessarily inoculate one against the effects.

The lessons for investors and traders alike is to view such comparisons with the same skepticism they would have for computer models.  If many others are influenced by the questionable information, the contrary trade is indicated.

Comments

[oldprof]

Hi Bernardo,

Thanks for your comment and the book pointer, that I'll add to my reading list.

The pervasive nature of this is why it is so important. If more investors realized what was happening, they could improve their performance.

Best,

Jeff

[Bernard Guerrero]

Common enough flaw. In my own line of work, I'm regularly bombarded by folks from other parts of the firm who think they've spotted some key to credit defaults. Even quite experienced underwriters fall prey to it. The problem, of course, is that they've never looked at a control group or formally back-tested "factor X", which often enough is just as or more prevalent in the non-defaulted population.

More generally, Philip Tetlock covers a lot of these sorts of bias in "Expert Political Judgment". Don't let the title fool you; his immediate subject matter is punditry on foreign affairs and macro-econ, but I think the basic lessons are widely applicable.