The Most Common Error in Stock Market Research

There is a very common research mistake. It is pervasive in Wall Street research, even that presented by the big-name firms. My academic friends are not immune, partly because they have their own set of incentives.

My mission in this post is fourfold:

1. Explain the problem in a way that it can be readily understood;
2. Show how you can spot it in practice;
3. Provide a clear example;
4. Suggest some other applications.

The Problem – Selecting the Right Data

If you take a course in research design, one of the first topics will be determining the right data and sources for your analysis. To help us stay open-minded, I will use one of my favorite approaches – a sports analogy.

Let us suppose that we wanted to predict the total points that would be scored in tonight's basketball game between Wisconsin and (The) Ohio State University. The game is being played as I write this, so the exercise is purely academic. Here are some possible choices for our analysis:

1. Take the entire history of college basketball and use the average as our forecast.
2. Use data only from the time since the shot clock was put in place.
3. Use data only since the three-point basket was introduced.
4. Use only data from the Big Ten, which might differ from other conferences.
5. Use data only from Ohio State and Wisconsin, who might have different team tendencies.
6. Use data only from these teams in the last few years, perhaps reflecting their current personnel.

Note that the data becomes more relevant as it gets more specific. Please also note that there is still plenty of data for our problem, since college teams play 30 or so games each year. Even a few years of data would provide 100 cases.

The Stock Market Comparison

Let us take what we learned in step one and consider how it applies to the stock market. Suppose that we wanted to forecast tomorrow's trading volume at the NYSE. Here are just a few of the major changes in stock trading (readers are invited to add more) since the 1792 agreement signed under a Buttonwood Tree.

• Stock quotes replaced a ticker tape.
• Securities regulation to provide information.
• Competitive commissions.
• The invention of computers.
• Options trading.
• Futures trading and arbitrage.
• Online trading.
• New NASDAQ rules and deep pools.
• Decimalization of stock prices.
• SOX and regulation FD.
• High frequency trading.
• Individual stock circuit breakers.

There are other elements, including the more active role of the Fed, but you get the drift. If you were interested in predicting volume, you probably would not use data that was more than a few years old. Too much has changed.

Even when it comes to more general market analysis I am not interested in what happened in the Taft Administration, the FDR era, or even the Ike years. I do not care much about the Nixon years, or even Jimmy Carter. We at least need to get to the modern era of an active Fed, active stock trading with low commissions, and broader access to data through financial television and computers.

An Example

For the purposes of this post I want to use a very innocent example from two of my favorite sources – both valuable contributors to our understanding of markets and current issues.

Let us first look at this chart from my friend Doug Short:

This is a beautiful chart. It is accurate and provides the most comprehensive history available from any source. Doug notes that the long-term, inflation-adjusted increase in stock prices is an annualized growth of 1.73% and that current values are 48% above this trend.

When I look at Doug's chart my eye does not follow his proposed regression line, mostly because I am totally uninterested in the old data. I imagine a different line, starting with the post-war period – surely more relevant. I also imagine a line beginning in 1982, where the data become even more relevant.

The starting point of 1871 is represented not because it is best, but because that was the earliest year for which Dr. Shiller could generate data. There is not a strong research reason for the choice.

While I was pondering this question and considering developing my own chart, I discovered this presentation from Scott Grannis:

Scott's chart is not inflation-adjusted, but it also does not include dividends. The conclusion is dramatically different, showing that stocks are in the middle of the long-term trend – growing at almost 7% a year plus dividends.

Neither source gives any particular reason for the choice of starting point – and that is my main focus here.

Great analysis begins with choosing the right data. Everyone has heard the expression "Garbage in, garbage out." This is where it starts.

Other Applications

If you understand this problem, you have jumped the first (and most important hurdle) in identifying strong research.

It will help you grasp the mistakes of most recession and business cycle forecasters. They simply do not have enough relevant cases to do a good job of ex-post analysis.

You can see the mistakes of those whose research identifies "bad times to invest." They also do not have enough past cases, so the inferences are unsound.

You can see the shortcomings of leading academics. They get respect for exhaustive and thorough analysis, finding data that others have missed. That is fine for their book reviews. You and I need to apply a higher (different?) standard. The popular book about why "this time is different" book has only a handful of truly relevant cases.

Investment Implication

The world wants "actionable investment advice." Fair enough. I have been acting on the principle described here for several years – with weekly articles to explain.

The basic conclusion is that many of the popular pundits, despite their apparent use of data, have developed inaccurate and over-fit models. It is better to have simple models with more relevant data. These may not seem as impressive at first glance, but prove to be more robust in practice.

More to come on this important theme…..

Comments

[Tim in Albion]

"by the time the average person saves a "decent sized investable account" (age 40+), they will be using it with in 20 years (retirement 60+)."

This is just silly. Investment begins when you start saving, and does not end until you stop needing money - long after retirement.

[annie mishu]

A very good blog post, I have book marked this internet site so ideally I’ll
see much more on this subject in the foreseeable future!
Thanks & Regards :

[hangemhi]

Miles, great comment until you mentioned Keynes. You see, it is those who interpret Keynes who are wrong, not Keynes himself. It's a shame that so many have supposedly debunked Keynes when all they have debunked is his wayward followers. Most of what you hear from a guy like Krugman is right, but he can't convince you because he is also wrong.... and you can prove he is wrong. Unfortunately even misguided keynesians like Krugman are still far, far closer to the truth than other "mainstream" economics like monetarism. The guys who get it right these days call themselves "monetary realists" because their contribution is recognizing what the various economic schools of today have simply made up vs. how things actually work. For instance, most economists simply remove the banking sector as too messy for their models. Not messy vs. accurate - you choose :)

[RB]

BTW, I forgot to add - you can reconstruct the total return chart and add a trendline at 6.57% annualized real rate of return which together with a long-term inflation of ~2.5% would represent 9% total return annualized. By my estimates, an S&P at ~1600 currently would lie on the median line. Needless to say, one could go above trend, at trend and below trend and hence has zero predictive value on future prospects.

[Stock Prices]

The

Stock Priceresult of these sales is a record revenue of $54.5 billion and a net quarterly profit of $13.1 billion, giving Apple a total of $137.1 billion in the bank. Last year’s quarterly profits were £13.06 billion, resulting in flat growth year-on-year, which may worry some investors and be the cause of the drop.

[RB]

Here's one more chart you could add to that list. The 2000 high doesn't look as bad as on Scott Grannis' chart.

[Miles Hoffman]

If you extended Doug's chart to the right (to today) OR extended Scott's chart to the left (even to Doug's starting point), BOTH would support the author's intention:

1) Scott's chart (and article) and his view is that the market is fairly valued (inference=it can be bought). His chart infers a "buy and hold" and I think it is absolutely wrong.

2) Doug's chart (and related articles) is focused NOT on the top "trend line" chart but rather on the variance from the trend shown at the bottom. His point is that there are cycles in the market that you need to understand and this is the exactly right viewpoint IMHO.

Crestmont Research (and Ed Easterling) is the best source and is the subject of my blog article that I think will be listed (but here it is):

http://mileshoffman.blogspot.com/2009/11/whats-your-decade-man.html

In short, BOTH charts shown cover LONG TIME PERIODS but the average person has, at most, only a 20 year investment horizon: by the time the average person saves a "decent sized investable account" (age 40+), they will be using it with in 20 years (retirement 60+). Thus they have a very short time period in which to invest (not Doug's 110 yr or even Scottt's 60 yrs).

Thus it is important to understand the "shorter-time" cycles in the stock market, which succintly, involves a cycle from single digit PEs to over 20x. We're closer to the peak of 20x, especially given record corporate profit margins, so be careful and shorter-term oriented.

This is where Doug's chart is critical (or Crestmont's data): The current market cycle is still, as far as I can tell, a SECULAR BEAR MARKET. As my blog points out (over 2 years ago and STILL applies), we're in a hell-of-a CYCLICAL BULL rally within the SECULAR BEAR (and a SECULAR BEAR "violently moves" down AND UP.

Keynes was absolutely wrong, but he got one thing right: "In the long run, we're all dead." (Save but) DON'T INVESTMENT for the long-term, it can kill you!

So for the average investor, NOW is a bad time to buy the market (unless they understand the shorter CYCLICAL bull and are NOT "BUY AND HOLDING").

[Johan Lindén]

Very good post!

Actually I'm studying data analysis now just to get these things better.

[Tim in Albion]

LOVE this. You explicate one of my pet peeves! (Not surprising, since I have so many; high probability you will chance upon one or more...)