Results Matter: Conducting Objective Reviews

Results matter.  Investors (and traders) seeking information need to know what to read.  Time is limited, so everyone depends upon those who provide "linkfests."  At "A Dash" we highlight the link sources that we regard as the best on the Web.

Keeping Score

Getting the links to many sources is extremely valuable, and we salute those who read everything and point out key stories that we all want to read.  Having said this, the daily links often do not tell the whole story.  For this we must look to those who examine the long-term records.

Market Predictions.  CXO Advisory (now added to our featured links) uses first-rate methods to analyze many questions of interest to investors and traders alike.  Often acting in response to questions and then studying the public record, CXO evaluates market predictions and provides a confidence indicator as well.  In recent days CXO has evaluated the "unimpressive" forecasts of Bill Gross and the "very poor track record" of Jeremy Grantham.  For a complete list of evaluations, readers can check out the Guru Grades section of the site.

Joining in the reality check is Eddy Elfenbein, often a source of information and research on specific market questions, as well as solid advice.  Eddy points out the error in media coverage of Robert Shiller, whose book serendipitously hit the markets at just the right moment.

Shiller is a perma-bear and investors who followed his advice lost out in a big way. In the last five years, the S&P 500 has doubled. Shiller missed that and every other rally. He was a bear long before the market nose-dived and he’s continued to be a bear ever since.

Economic Forecasts. Bill Rempel has a well-deserved reputation for  hard-hitting analysis that is both accurate and fair.  His work has achieved great readership even without any acknowledgment from the big-time bears.  It is on our daily reading list and should also be on yours.  His recent series of posts took a careful and well-researched look at the economic predictions from Barry Ritholtz, Mish, and Nouriel Roubini.

All of the people reviewed write leading blogs and get massive media exposure.  They are all extremely intelligent and articulate.  They write persuasively and well.  Those reading their work find it persuasive.  Reviewing their overall record is hard work.  It would be a service to the investment community if  the gatekeepers of the investment blogging world --blog aggregators, journalists, and the linkfest professionals-- give the attention due to those doing comprehensive reviews.

The Sources of Error

The spectre of confirmation bias confronts everyone studying the economy and the stock market.  It is so easy to make two important errors:

1. Citing data selectively, and
2. Interpreting data with a spin -- perhaps unintentionally.

Since all evidence is imperfect, this happens readily.  Picking two sources of commentary on Friday's jobs report (it could have been any of the listed guru's)  Accrued Interest writes a thoughtful commentary suggesting that there is "lawyering" going on. 

Meanwhile, there was sure a lot of lawyering going on across the blogosphere. One of the reoccurring themes here on AI is that too many investors act like lawyers, arguing their case from a predetermined point of view, and seeking out facts that support that point of view. Good investors act more like detectives, starting with an open mind and letting the evidence lead them to a conclusion.

Nouriel Roubini and Barry Ritholtz's posts on yesterday's NFP release both smack of lawyering. Both harped on the large increase in teaching/education jobs, which is a legitimate albeit limited point. If changes in the school calendar made measuring teaching jobs difficult, then indeed July and August reports were understated. So the bad news we thought we were getting in those months really wasn't as bad as we thought. I note that neither blogger wrote one word about the education job issue after either the July or August report. (Here's Roubini and here's Ritholtz here and here.)

In both cases, I can't help but feel as though the writer is looking for bearish news to report. When August jobs was reported as negative, both sounded the alarm that a hard landing was coming. But when August was revised higher because of a legitimate discrepancy in the data both bloggers dismissed the revision as a mere data discrepancy. You can't have it both ways. The truth is that the job situation is better than we had thought.   

Friday's non-farm payroll report (NFP) is indeed a great example.  Those with a bearish economic view have engaged in a multi-year trashing of the Bureau of Labor Statistics.  The arguments range from conspiracy theories to accusations of poor methodology.  The rhetoric is loaded with phrases like "phantom jobs" with emphasis on the BLS implementation of a means of recognizing new workplaces, the birth/death adjustment.

The simple fact is that the birth/death adjustment dramatically improved the jobs forecast for the last period measured, as it has for all prior periods since it was implemented.  Nearly 80% of the jobs projected through this adjustment proved to be real, but the bearish pundits focused on the 20% error.  As regular readers know, the NFP report is a major interest of research at "A Dash" .  We plan a more comprehensive review of the multi-year performance of BLS job measurement.

Meanwhile, readers might wonder when these pundits will start with the data rather than their conclusions.

Comments

[VennData]

The limits of experts theme is an essential because it gets investors away from absolutes. Absoutes make investors vulnerable in a field where the free lunch of divesification protects you from making all or nothing bets.

The list of absolutists is long. We easily spot the ones we disagree with philosophically, politically, or even simply owing to style.

Collecting the data is easier said than done look at the recent Barron's / Cramer tiff over his record. There's no reporting standard for CNBC's or Bloomberg's pundits past results, and as for soon-to-launch FOX business news with 'less jargon' I'd like to know which words they're planning on droppimg! ...and in the land where 99% of newspapers carry horoscopes... rules or regulations on printed or digital media are at least a generation away.

I do believe that even though the Fed has so many experts at their disposal, they ultimtely serve at the discretion of Congress so say what you will about their decisions, politics has some influence on them.

For me, I don't care how low someone claims they'll cut my taxes or cut government spending, when they hold Creationism as equivalent to the Theory of Evolution I am very suspicious of what other of their absolutist beliefs effect their decisions.

[Macro Man]

Given that Bill was kind enough to link to my site (both here and on his own), I thought I'd share a few thoughts.

Call me cynical, but I generally believe that most people who simultaneously provide commentary and have something to sell are primarily concerned with looking intelligent, so as to sell more (funds, website subscriptions, newsletters, etc.) For many people, there seems to be a fear that being wrong (or admitting that they might be wrong) undermines their perceived "authority", and thus the amound of funds/subscriptions/newsletters that they can shift.

IMHO, the opposite is true, and admitting that one is wrong, not just in a trade but in an underlying premise, is a sign of both confidence and intelligence.

An additional factor is that making outrageous statements/forecasts is away of attracting notoriety....and if you've got funds/subscriptions/newsletters to sell, there is no such thing as bad publicity.

I suspect that each of these factors is behind the behaviour of some of the individuals highlighted in the post. Of course, the same can also be true for Wall Street economists/strategists as well (Messrs. Roach and Rosenberg come to mind, as well as a host of lesser-known lights) who don't necessarily have a specific product to sell.

Overall, however, I overwhelmingly agree with this post...which is why I find myself reading less and less research these days...the facts are tailored to the conclusion, rather than the other way around.

(Disclaimer: I was briefly a colleaguie of Paul McCulley's nearly a decade ago. He was one of the smartest guys I've met. It really makes me wonder, therefore, how PIMCO's Fed calls could have been so god-awful for the last several years, other than a desire to sell more bond funds.

I once wrote a post called Nouriel Roubini is a Big Fat Idiot, so my views with respect to him are probably skewed. I have a very good relationship with Brad Setser at RGE, however.

Poster comments made on my site once prompted Barry Ritholtz to admit bias, so like Carl Spackler, I guess I got that going for me. I may, however, not have an unskewed view of Mr. Ritholtz' value added.)

[oldprof]

Mike --

If you read the CXO piece, it says that Grantham's funds have performed fairly well and acknowledges that many of his forecasts are of the long-term nature. They suggest that his stock picking may be better than his market timing. I like the fact that CXO describes the limits of the information they have and their methods. They provide enough data for a reader to verify the conclusion (or not).

One of my smartest clients swears by Grantham's work. He does just what you do, reading everything and he is aware of the time frame.

A continuing theme here is that you need to recognize expertise and the limits of expertise. The people in today's examples are all highly-respected experts with an important message. They have not done well at market timing. The guru grading helps consumers like us to use experts more effectively.

Thanks for another thoughtful comment. Maybe I can figure out how to say this better in the book.

Jeff

[RB]

I like CXO, but in the spirit of confirmation analysis, I find nothing first-rate about their guru analysis. Readers should check for themselves.

[Mike C]

"In recent days CXO has evaluated the "unimpressive" forecasts of Bill Gross and the "very poor track record" of Jeremy Grantham. For a complete list of evaluations, readers can check out the Guru Grades section of the site."

I follow Jeremy Grantham very closely. I read all of the quarterly letters along with their monthly updates on forecasted returns for various asset classes, as I use GMO's work to aid in my own decision making.

I'm not sure how exactly CXO is evaluating him, so I am highly skeptical of their conclusion vis a vis him. When I have time, I'll have to investigate further. I do know that Mr. Grantham does not make short-term market forecasts but instead projects long-term (5+ year) returns. On that count, I know he is accurate because I have been following them for awhile. There is a fine line between being early and being wrong that isn't possible to draw with 100% accuracy.

I don't remember where I read this, but the statement was successful investing is about getting the big things right, and if you get the little or moderate things wrong (even frequently) it doesn't really matter to your returns.

Whether the market is going to be up 5% or down 5% in the next 3-6 months is a little thing, close to irrelevant. Big things are things like avoiding Internet/tech/large-cap growth in 1999 and buying REITs and small-cap value in 2000. Grantham was pounding the table on both of those themes at the time. Anyone who listened came through 2000-2002 smelling like a rose.

Presently, Grantham is saying to avoid small-caps, and focus on large-cap high quality. Whether that works over the next 3 months really isn't important unless you are a hyperactive trader. What's important is if that is the right call for the next 5-7 years and we won't know that until 2012.

[Bill aka NO DooDahs!]

I seriously considered doing a comparison of Ritholtz' coverage of the job report to Macro-Man's coverage of the same. It would have been very instructive.
http://macro-man.blogspot.com/

Barry's biggest problem as a fund manager is confirmation bias. In his interview on Bloomers in early August, he described himself as a "miserable bear" who had been long due to his quant models, and who had to field countless calls from clients who said "you sound so negative and yet we're full up with stocks!" Go to this post of his
http://bigpicture.typepad.com/comments/2006/12/reality_check.html
and read the final comment from Barry.

I'll wait.

Back? OK.

So the client accounts that HAD to be long outperformed the index, because they were just picking stocks based on his quant models. Meanwhile, his fund lagged the index and missed the better part of the rally, because it was damaged by his CONFIRMATION BIAS - which in turn was predisposed bearish based on his flawed economic analysis.