A Dash of Insight

ISM Interpretation: Watch that Decimal Point!

Today's ISM number for May came in at 49.6, slightly below the level indicating expansion in manufacturing, but a touch better than expectations.  Is the ISM report important?  We'll look first to news reports and pundits, and then provide our own take.

News Interpretations

Bloomberg gave a balanced account, noting economic forecasts as follows:

Economists forecast the index would decrease to 48.5 from 48.6 in April, according to the median of 75 projections in a Bloomberg News survey. Estimates ranged from 46 to 50.5.

and also...

ISM's gauge of new orders increased to 49.7 from 46.5, while a production measure rose to 51.2 from 49.1, ISM said.

The Wall Street Journal's Real Time Economics has the usual nice collection of economic reaction.  The title, "Economists React: ‘Severe Recession Has Been Averted’?" captures the main theme.  The economic data have been good enough to cause many to reconsider their recession predictions.  Read all of the comments for yourself, but the consensus indicates a greater chance of Q2 GDP at a rate of 1.5% or so.

Significance of the Report

At "A Dash" we always wonder how economists can make forecasts of survey results like this one or of consumer confidence.  The analysts at Briefing.com share this concern.  The following describes their concern:

This is a highly over-rated index.  It is merely a survey of purchasing managers.  It is a diffusion index, which means that it reflects the number of people saying conditions are better compared to the number saying conditions are worse.  It does not weight for size of the firm, or for the degree of better/worse.  It can therefore underestimate conditions if there is a great deal of strength in a few firms.  That may well be what is happening at present with exports booming at large firms, but not necessarily across all manufacturing sectors.  The current readings on the ISM manufacturing index are providing a more negative view of conditions than the actual industrial production data.  The data have thus not been either a good forecasting tool or a good read on current conditions during this business cycle.  It must be recognized that the index is not hard data of any kind, but simply a survey that provides broad indications of trends. 

We see the ISM data as a contemporaneous economic indicator that we analyze along with payroll employment.  We have frequently warned in the past when we expect a surprising negative result with market impact.

To summarize, this was good news, showing an economy that is growing below trend but defying the recession predictions.

An Alternative Viewpoint

Over at The Big Picture, one of our featured sites, Barry Ritholtz, the self-styled gonzo economist, had a different view about what we should see in this report.  He pounced with the "Bad Headline of the Day."  His point was that the article called the report a gain in manufacturing activity, which it was not.  He did not, however, mention that the report implied a higher GDP than was expected.

We congratulate Barry on noting the decimal point error of the headline writers, but we are left wondering about the main story.  Does he believe that this was bad news?  Also, this is a survey.  Is he considering the confidence interval and statistical significance?  Can we be confident that the null hypothesis of "Not 50" can be rejected?

Big Picture Reader Contest

The quibble over the decimal point created a mini-contest in our office.  We started arguing about data interpretation on The Big Picture.  We have recommended this site since our inception, and we read it daily, usually with great care.  We also watch Barry on TV, as do our clients.  Sometimes we dig into the archives, usually when we are trying to answer a question like "What did people think was wrong with the market in June, 2004?"  The answers are all there.

But back to the contest.  We offer a prize to the reader who can find an article on The Big Picture where Barry analyzed the data and suggested that the results were more positive for the market or the economy than the official report suggested.  The prize goes to the most recent entry.  It must be a regular report, of the sort listed on the Briefing.com calendar, where Barry's interpretation was more bullish than the report seemed to indicate at face value.

Conclusion

Each economic data point shows an economy slogging along below trend, at great cost in terms of lost potential, but not as bad as many expected.  Today's economic numbers were pretty good, but were overwhelmed by the S&P decision about future writedowns for some investment banks.

We shall look at the payroll employment report forecast later this week.  Meanwhile, our indicators respect the tape.  We are more cautious in the intermediate term.

Cherry Picking in Data Analysis

There is a type of research that is especially dangerous for individual investors.  The researcher takes current data and makes a statement like one of the following:

• If you avoided the ten worst trading days over the last five years....
• If you missed the ten best trading days over the last five years....
• If you threw out the ten strongest earnings reports....
• If you threw out the ten worst earnings reports...
• If you avoided the last five recessions....
• If you threw out the companies with the strongest stock performance...
• If you threw out the companies with the weakest stock performance...

This type of analysis is pretty easy to do for anyone with a computer and a data set.

Many trading systems do backfitting, avoiding the recessions or downturns.  It is easy to find an indicator that gave a definitive signal when looking at past data.  The problem is that such systems, lacking rigorous development of hypotheses, failing to use out-of-sample data, and willing to accept an insufficient number of cases, usually produce post-diction rather than robust predictive models.

The Current Example

Barry Ritholtz at The Big Picture highlights an interesting situation posited by Mike Panzer.  Mike reports that a small number of stocks have powered the Nasdaq higher.  Mike  concludes as follows:

Finally, 13 out of 100 stocks -- 13% -- are responsible for two-thirds of the overall advance.

While this heavy lifting by a small number of shares does not mean the index can't go higher still, history suggests rallies that lack widespread participation sometimes lack long-term staying power.

What to Conclude?

We are troubled by this facile conclusion, which was reported without comment by various pundits.   We enthusiastically endorse the more analytical questions  raised by Barry:

What might this mean?

Are Technicals Waning as a Positive Influence? I'm not exactly sure -- What I'd like to see is how past rallies have moved forward in terms of leadership.

Is it unusual to have 13 stocks in the NDX’s 100 account for 67% of the aggregate advance? Is this unusually narrow? When has this occurred, early or late in a run?

I don't know the answers to these, but I am curious . . . 

In scholarly research one would not start with a conclusion, but with a hypothesis.  It might go something like this.......

When fewer than fifteen stocks in the Nasdaq account for two-thirds of recent gains of X percentage, we define the leadership as "narrow."  Looking back on the Y number of cases fitting this description, we note that stock returns over the following Z days were as follows (table included).

Even with such a statement there are issues about how the definition of narrowness was determined, whether there were enough cases, and whether the researcher really generated the proposition and then tested or just used all of the data to identify key parameters.  This approach would at least provide some comparison.  Ideally, the relevant data would be provided to other researchers to check the selection of parameters.

In the absence of such data, one is left with questions.  The indices are weighted by capitalization.  What is the performance of the rest of the index?  What is the market cap percentage of the top stocks?  Markets often look for leadership.  Is a gain by some of the top stocks indicative of success or of failure?  If other stocks are lagging, is it possible that the market will later show strength in the laggards?

A Final Word

Like Barry, we do not pretend to know the answers to these questions.  In our effort at "A Dash" to raise the standard of Street research, we try to highlight certain problems.  Often these are conclusions that are readily embraced by those who seek support for what they already believe.

Some research is driven by conclusions, not by hypotheses.  It is not scientific.

As usual, the discriminating investor, insisting upon strong research methods,  can gain a contrarian advantage.

Mortgage Availability and Personal Consumption: The Dubious Relationship Breaks Down

What happens to Personal Consumption Expenditures when banks tighten up mortgage lending standards?  With the release of April PCE data, we can now check out a dubious, but widely publicized prediction.

Background

Two months ago the influential Doug Kass repeatedly published a chart that purported to show a near perfect correlation between lending standards as measured in a quarterly Fed survey and year-over-year PCE.  The chart was also posted and discussed at the widely read Calculated Risk blog site.

In a three-part series on "A Dash", we went to the original data sources and performed our own analysis.

Part One. We showed readers that the correlation was exaggerated, that the scale had been manipulated in a deceptive fashion, and that the causation argument did not fit the data.  The existing correlation was probably spurious, the change in each resulting from changes in prevailing economic conditions.

Part Two.  We explained the difference between a confidence interval and a correlation, helping readers to understand the difference between substantive significance and statistical significance.

Part Three.  We compared the chart to various optical illusions, showing why accurate measurement is better than the eye.

The Prediction and the Result

The original Kass chart showed an ominous tightening in lending standards, suggesting that we were on the verge of a similarly precipitous decline in personal consumption.

As one can see from the updated chart, (the new segment indicated by a red line around it) the PCE change was modestly lower.  The rate of PCE growth continues well within the range from the last two years.

The Kass prediction was incorrect.

Conclusion

As we concluded in our original analysis:

The current mortgage market is much different from that of 1990. Nearly everyone -- especially Doug Kass -- believes that recent standards have become quite loose.  Some tightening is a logical reaction.  There is nothing in the data he presents that shows that this should result in a decline in consumer spending.

Such a decline might occur, of course.  There may be other reasons to link housing and the economy.  That is beyond the scope of Doug Kass's argument and this response.  The point here is that the data in the original Kass chart do not support his conclusion.

The next Fed survey of bank officers should be reported any day, so there will be a new data point to add to the chart.

A History Lesson for Wall Street Resesarchers

Wall Street researchers are terrible at history!

The worst examples occur when they make hasty and sloppy comparisons.  A current example is finding an old chart from one time period and lining it up with current trading.  This is a blatant form of data dredging, but some find it to be compelling reading.  (Here is a typical example from The Big Picture, but please note that the inference is not endorsed by Barry Ritholtz).

The problem is that everyone who follows the markets looks at charts on a daily basis.  Even the non-technicians claim some basic skill.  Show us a chart, and we all have an opinion.

At "A Dash" we have tried to point out that the data we really need for good research is rarely available.  This forces us to reach back in time --- often way back -- to get enough data to review more than one or two market cycles.

We seriously doubt that market data from the Taft Administration has much relevance for today.  In fact, we question any information from the era before stock options and futures, and maybe before the invention of the personal computer.

There are some market characteristics that endure over time.  Sure, everyone should read and enjoy Edwin Lef́aevre's Reminiscences of a Stock Operator (on our recommended list).  The question is whether the inferred psychology from this era is equally applicable today.

It is a question that must be asked repeatedly, every time someone presents data that reaches back more than twenty-five or thirty years ago.

It helps to be fully grounded and in touch with what people really believed in the era you are using.  If one wants to use data from 1900, why not see what people were thinking then?

There is a wonderful list of predictions for the year 2000 made in 1900 by J. Effreth Watkins in The Ladies Home Journal.  It is a rather long list, but very entertaining.  Each of us will find certain favorite observations.  (Thanks to Tyler Cowen at Marginal Revolution, where we spotted this).

Our own overall conclusion is that Watkins was very good at extrapolating technology known at the time -- the rise of automobiles and improvements in medicine (including diagnosis), and the anticipation of FedEx (with a lot of pneumatic tubes).

He was less successful at quantification.  The U.S. was to have 350 to 500 million people.  We would have a life expectancy of 50 instead of 35 in those days) and everyone would be able to walk ten miles a day or be thought a "weakling."

He was predictably weak on technology that was just appearing like airplanes, which he thought would never replace the automobile for passengers and freight.

Most mysterious are the predictions that must have seemed plausible, but went terribly wrong?  Why DO we still have all of these mosquitoes?

The lesson for those doing market research:  Take care to make sure that historical comparisons are well-researched and appropriate.

The lesson for our fellow consumers of Wall Street research:  Let the warning bells go off when there is a chart or analogy including a time period from long ago.

Quantifying the Economic Impact of Housing Declines

Everyone wants to know about housing and the economy.  Is the sub-prime lending issue something that will extend to other mortgage classes?  How much might home prices fall?  Will the impacts lead to reduced consumer spending and an even greater economic effect?

Doug Kass, the popular hedge fund manager noted for his skill in short selling, has been one of the leading voices on this topic.  His view is that this is a major economic problem and one that will affect the home prices and portfolios of individual investors. 

Writing today in Street Insight, the valuable premium service of theStreet.com, Doug does a "back-of-the-envelope" calculation of the likely housing effects.  At "A Dash" we are big fans of such an approach.  Trying to think about quantities and actual impacts is essential.  Market participants often have knee-jerk reactions to news, mostly because they cannot estimate the real effect.

To get Doug's entire argument, you need to be a member at theStreet.com, but they might republish it later for the general public.  Since he is such a frequent and popular TV guest, you will also see it soon on CNBC.  I am quoting just one section, the place where he quantifies the likely impact of what we all read about every day:

Quantifying the Impact of Tightening Credit
I would conservatively estimate that about 55% of the subprime borrowers, 25% of the Alt-A borrowers and 15% of the prime mortgage lending borrowers will no longer be able to secure financing for new homes because of tightened conditions. (This will produce about a 25% drop in housing demand). Speculators and investors - who were responsible for nearly 20% of all home purchases in 2004-06 - will also find it more difficult to secure borrowings and it is likely that this buying category will revert back close to their historical demand role of about five percent of all homes. (This will result in another 10%-15% drop in housing demand). Finally, end of economic cycle conditions (lower consumer confidence, slowing economic growth and moderating job growth) should contribute to another 10% drop in housing demand - as it has done historically. In total (adding the above three influences), new home demand should fall off by almost 50% (vs. the rolling 12- month average showing a 17% drop off in 2007) - even before the effect of a market inundated by record foreclosures is considered.

What is Wrong?

This estimate will sound frightening and persuasive to most who hear it.  For students of economics, the problem leaps out.

A statement like "a 25% drop in housing demand" has no economic meaning.

Non-economists speak in terms like Doug.  Someone is either in the housing market or he/she is not.  A home is either on the market or it is not.  Completely lost is the concept that a buyer may not qualify for a loan of one size, but will qualify for a smaller loan.  Homes for sale at one price are withdrawn from the market if the price is lower.

The century-old concept  is shown in the supply-demand curve that one sees the first day of Econ 101.

As price declines, the quantity demanded increases.  As price increases, the quantity supplied increases.  The actual exchange price clears the market at the  quantity where the curves intersect.

When one wishes to describe underlying changes in either supply or demand, these are characterized as "shifts" in the curve.  Reduced demand, for example, shifts the downward-sloping blue curve to the left, meaning that there are fewer sales at a lower price.  Reducing prices shifts the supply curve to the right, increasing the quantity.

Is this News for Doug Kass?

Readers should understand that Doug Kass knows everything we have written here.  He has an MBA from Wharton and he certainly studied economics.  He knows full well that one cannot describe supply and demand as a binary function -- either in or out.  One of the first things you learn in economics is that shifting the curve has a much smaller price effect than one would think at first.  So why is he writing this?

Conclusion

We wish we could provide a good answer about the impact of housing and mortgages on price and quantity, but we cannot.  To do that, one would need some data about the shape of both curves, at least enough to make an estimate.

We can say with confidence, however, that the Doug Kass scenario is extremely unlikely  The relevant curves would need unusual shapes and make massive shifts to have the impact that he forecasts.

Mortgage Availability and Personal Consumption: Round Three

Background

During my vacation, Doug Kass posted again on the topic of housing and personal consumption.  His analysis can be divided into two parts.  Part one "replied" to our analysis and part two elaborated further his reasoning about the connection, the causal model, and what he expected to happen.

Here is what he wrote:

Many, like "Mad Money's" Jim "El Capitan" Cramer, "Kudlow & Company's" Larry Kudlow and others, readily dismiss the potential spending consequences of substantially less capacity in the subprime mortgage lending market and the emerging trend by mainstream originators and lenders to reduce lending in the primary mortgage market and for refinancing cashouts. Indeed, Jim takes the subprime issue one step further, noting that the mortgage house of pain will have a salutary market and economic result, as it will hasten the Federal Reserve's path toward monetary ease. Shockingly (at least to me), many others can't comprehend the link between mortgage availability and consumer spending, claiming that the correlation between the two variables (seen below) is unclear.

Those wishing to see the entire statement can look here and here, but a subscription might be required.  I have quoted the section relevant to our work.

As you can see, "A Dash" is the unnamed "many" in Doug's statement.  Well, we have been in worse company than Kudlow and Cramer!

As purported refutation of our analysis first posted here and with more information here, Doug merely posts the same chart again, with no effort to respond to our reasoning.

There is a serious flaw in Doug's analysis, and it plays upon the perceptions of both traders and individual investors.  The concept of correlation has a measurable statistical basis.  It is not a matter of opinion. It is a fact.  We believe that the chart is an optical illusion.  Here is a familiar example, one that most readers have probably already seen:

You can either use your eyes to tell you which line is longer, or you get a ruler and find out the truth.

Readers looking at the Kass chart see with their eyes a strong correlation.  This is because the key elements of the chart have been pushed together to emphasize two key points.

To our surprise, readers at Calculated Risk commented both on our site and theirs that the areas we described as "no fit" in our chart actually showed a strong correlation.  Take a look again at our "no fit" chart.

Readers saw what they called the "Batman" segment as quite similar.  They also were undisturbed by the periods where the two lines ratcheted along, out of phase.  This is a trick played by the human mind, the most powerful computer.  Our minds take two patterns and look for similarity.  Sometimes that leads us astray.  Correlation means that the movement in one variable corresponds in both direction and magnitude to the other variable--and at the same time.  The fact that the two lines both have "Batman" characteristics is unhelpful to the trader unless the movements directly correspond.

Since this is difficult to see visually, it is important to use statistical techniques to measure correlation.

An Aside

To put this in the proper perspective, we should remind readers of our mission at "A Dash."  We review and analyze Wall Street Research where we find interesting errors.  We have no shortage of candidates.  We do not have a mission linked to a particular market viewpoint.  We choose our market stance based upon the evidence we see.

In the Doug Kass case there is a proposition that historical data show a strong correlation suggesting that personal consumption expenditure declines are imminent.  Using traditional statistical methods, we find that the data cited do not support this conclusion.  It does not mean that Doug is wrong in his ultimate conclusion -- just that his frequently-cited chart does not support his viewpoint.

In the areas of the chart that we cited as "no fit" the correlation is much lower or non-existent.  The entire strength of the chart depends upon a few data points.  If readers do not see this, then we have failed in our mission to educate about the danger of visual chart interpretation.

Going Back to the Classroom

There will always be some who make up their minds first and then see everything as evidence.  We cannot reach them.  It was surprising to us how many readers insisted on the correlation, based upon this chart.  What were we seeing that they were not?  To improve on our analysis in the first two posts on this topic, let us turn back to the yellowed notes of classroom instruction from decades ago.  To see why this is not a  strong or useful correlation, one must start by understanding what that means.

The concept of correlation is based upon a linear relationship between two variables.  With modern software, it is easy for anyone with access to the underlying data to calculate the correlation and statistical significance.  But that is not enough!

The concepts are from an undergraduate introductory statistics course where correlation and regression are two of many topics.

We present here four different data sets, all showing the same statistical characteristics -- averages, standard deviations, slope coefficients, etc.  The dramatic demonstration for students was that it was not enough to look at the numbers, one also had to consider a scatterplot of the data.  If the data did not fit a linear model, the entire concept of correlation was called into question.

In only the first case is the regression equation a good specification of the model -- a way of saying that the description and correlation really reflect the data.  In the case where the relationship is curvilinear, this tells the researcher that there is a missing independent variable.  The bottom two cases show a very strong relationship that is distorted by the presence of a single outlier, an unusual case deserving further study.  It is quite obvious that the regression equation is not a good description of the data in the bottom cases.

In real life, examples are never as clear as the pure cases from the classroom, but the Doug Kass chart comes pretty close.  We showed, in the chart repeated below, the scatterplot for the hypothesized relationship. 

When the optical illusion of the time series is stripped away, the reality is clear.  This is mostly a cloud of data with a few outliers that have a strong effect on the equation.  It is not really a good candidate for a linear model, and that means that the term "correlation" is meaningless.  (Those who really want to understand this should also read the prior two articles on the topic.)

The Real Conclusion

The data show that for a few quarters in the early 90's a decline of over 15% in mortgage availability occurred at the same time as a decline of about one percent in personal consumption expenditures.  There was a recession that probably caused both effects.  The data include no other examples of big declines, and the smaller moves are all part of a data cloud.

If someone told you this in words, you would not find it very persuasive.  Somehow the chart has an overpowering visual effect.  Do not bet real money on what you think you see.

Mortgage Availability and Personal Consumption -- An Update

Calculated Risk, a site on our regular reading list, has a story today discussing the same chart we analyzed yesterday.  (Thanks to reader "RB" for pointing this out).  Here is their conclusion:

We can be 99% confident that the YoY changes in real PCE are positively correlated with loosening mortgage lending standards.

Some observers are interpreting this statement to mean there is a .99 correlation -- a very different thing.  (Perhaps this is also what Doug Kass meant in his comment on CNBC.)

The data presented have a correlation of .43,  evidence of some relationship but far from a .99 correlation.  In a .99 correlation, the scatter plot shows every point in a straight line.  This is far different from the plot we showed yesterday.

This has a few outliers that define the relationship and a data cloud.  Any good analyst looks at a scatter plot when doing regression and correlation.  Looking at the residuals is the only way to know if a linear model is appropriate.  In this case, the warning flags should go up.

A correlation of .43 means an r-squared of .19.  The statistical interpretation of this is that about 19% of the variation in one series, as defined by the squared deviations from the regression line, is "explained" by the variation in the other series.  This is how one looks at substantive significance -- whether the relatinoship is important.

So where does the .99 come in?  A linear regression analysis calculates a slope coefficient and an intercept.  In this case, the slope coefficient for the equation is .10, as we noted on our chart.  This means that every 1% change in the mortgage availability measure is associated with a one/tenth point change in year-over-year PCE.  The slope coefficient has a standard error, calculated from the number of cases and the degrees of freedom.  The standard error for the slope coefficient is .02.

Since the coefficient is much larger than the standard error, we can be 99% sure (making some other assumptions about the typicality of the data we have) that the  "true" slope coefficient is not zero.    This is a test of statistical significance.

The confusion of statistical and substantive significance, and which measures are used for each, is one of the most common mistakes made by those without a strong background in research reseasrch methods.

To summarize, the Calculated Risk statement that we can be 99% sure of some relationship between the two variables is correct.

Everything in our article yesterday is also correct.  The degree of association is not as strong as the misleading graph suggests.  The entire relationship rests upon something that happened for a year or so in the early nineties.  The measure of mortgage availability is not very good for the purpose.  There is not enough data.  The resulting relationship is probably spurious.

As someone who taught these classes at the graduate level, I have no illusion that the average reader is going to appreciate these distinctions, however important.  It is a good illustration of how easy it is to be fooled by the eyes, and how difficult it can be to reach the truth.

Mortgage Availability and Personal Consumption: Some Dubious Evidence from Doug Kass

Doug Kass is on a mission!  He has posted the same chart four times in the last ten days and mentioned it even more often.  He has also cited it on CNBC appearances and probably other places.  Here is the key point, drawn from his column today (subscription required), entitled "The Worst Is Yet to Come: A Second Downleg in Housing":

I have tried to document the clear relationship of mortgage availability to personal consumption expenditures, which occurs in every cycle (up and down). You simply can't deny this relationship.

He sees the following chart as powerful evidence of some forthcoming bad news.  [click on all charts to enlarge.]

At "A Dash" we shall take the other side of Doug's mission -- since he is so powerful, let's call it Mission Impossible.  We are going to show that the evidence he presents should not persuade one of his conclusion.

Some Background

Doug Kass is the highly influential writer of The Edge on Street Insight, theStreet.com's useful product aimed at market professionals.  Doug also gets plenty of exposure on CNBC, in Barron's, and other places.  He modestly states that this is because there are so few bears left.  In fact, he has an appealing delivery both on television and in writing.  He is not just another loud, arguing voice and is quite persuasive.  Personally, I read everything he writes, but I use his information carefully.  Somehow he has been successful in managing his fund while being on the wrong side with his economic and market forecasts for several years.  How does he do it?  He does a lot of trading against his overall short position, makes some great calls, and takes profits on shorts when he is right.

The Chart -- A First Impression

When one wishes to make a convincing argument, adjusting the scales of a chart to fit the most visible features makes a powerful impression.  The Kass chart is scaled to emphasize the two "V" troughs and the current plunge in mortgage availability.  Whoever did this chart was familiar with Tufte's classic work (featured in our reading list).

An Objective Look

An expert view of the Kass chart would include looking for areas of no fit as well as the "V" troughs and also choosing a scale level that reflected the overall patterns, without undue emphasis on any feature.  With the supporting help of Renae, we found the underlying data and produced the alternative views.
This is exactly the same chart with a different visual emphasis.  The emphasized areas of "no fit" show that there is often no "relationship".

Another approach is to allow the Excel defaults to choose the scales, instead of trying to emphasize the two troughs.  Here is what one finds.
While the general shapes are similar, the discrepancies are very obvious.

Regression and Correlation
In his Kudlow appearance, Doug claimed a very high correlation for this relationship.  (My TIVO window scrolled off before I could replay it, but it sounded like he said .9).  Let us examine the regression equation, the correlation, and the corresponding scatter plot.
The overall regression equation has an r-squared of .19, meaning that one variable "explains" about 20% of the variation of the other.  Nearly all of the explanatory power comes from the handful of data points on the left -- the outliers.  Without these points there is little relationship -- a data cloud.  This suggests the need for examining the influential cases.

Looking at the Data

At "A Dash" we have maintained the importance of considering the variables in a relationship.  We insist on asking what is being measured, how good the measures are, and specifying a causal model.  We also have argued that it is important to select the right time frames for analysis, seeking enough similar cases from history, while not reaching back too far.  Let us apply these tests to the Kass bivariate relationship hypothesis.

The alleged dependent variable is personal consumption, taken from the Department of Commerce.  The causal variable is availability of mortgage credit, taken from a survey conducted by the Fed.

The Fed survey included 55 banks, so a shift of ten banks to "somewhat tightened" created the 18% decline showed in the chart.  No bank officer reported a "substantial tightening".  Moreover, it does not indicate whether standards were already tight or loose -- something that relates to size of mortgages as well as applicant qualifications.  Here is the key part of the question asked:

If your bank's credit   standards have not changed over the relevant period, please report   them as unchanged even if the standards are either restrictive or   accommodative relative to longer-term norms. If your bank's credit   standards have tightened or eased over the relevant period, please   so report them regardless of how they stand relative to longer-term   norms.

The result is that we know little about whether the current standards are high, medium, or low by historic norms.  We know that ten banks somewhat tightened mortgage standards, none substantially, and forty-five did not.

The Time Period

Kass asserts a relationship that occurs in "every cycle".  The problem is that his data reports only one "cycle".  The reason is that the Fed survey only goes back to 1990, so this is all of the available data.  We do not know about any history before 1990.

Instead of a powerful chart showing many cycles, the proposition is much simpler.  In the early nineties banks became somewhat more restrictive in mortgage lending at the same time personal consumption was declining.  The question confronting those examining the data is whether the current situation is similar to exactly one prior case.  That is all we have.

The Causal Model

When asserting causation, it is important to specify the causal model and to explain the relationship.  We are mystified by Doug Kass's assertion.  If a bank is more reluctant to grant a fresh mortgage, how does that affect consumption?  The potential buyer remains a renter, or continues to live in an existing home.

If the argument is that refinancing is more difficult, we have a different problem.  There are no data on that subject.  It also does not speak to existing lines of credit, or how those might be used.

Most importantly, the mortgage variable does not lead the consumption variable.  The Kass chart shows a simultaneous change.  As we have pointed out in a typical example, this is often the case when there is a spurious relationship.  Simply put, this means that some third (unspecified) variable is the underlying cause of two different effects.  If one merely looks at the effects, there is some apparent correlation.  In 1990, a weakening economy probably caused both effects.

Briefly put, the relationship in the one relevant prior case (a few points in the early nineties) shows two variables that are a result of something else.

Conclusion

The current mortgage market is much different from that of 1990.  Nearly everyone -- especially Doug Kass -- believes that recent standards have become quite loose.  Some tightening is a logical reaction.  There is nothing in the data he presents that shows that this should result in a decline in consumer spending.

Such a decline might occur, of course.  There may be other reasons to link housing and the economy.  That is beyond the scope of Doug Kass's argument and this response.  The point here is that the data in the original Kass chart do not support his conclusion.

CNBC Story on Market Correction: A Good Example and a Big Error

This afternoon CNBC's Street Talk featured two analysts discussing the prospects of a market correction.  The topic is both timely and important.  The participants were analysts using two different methodologies.  At "A Dash" this is the kind of segment that has us getting some popcorn and pulling up our chairs.

Hosting the segment was Erin Burnett, whose star is rapidly rising at CNBC.  Her strong educational and work background helps her to be a good interviewer.

What Happened

The first to speak of her two guests was Mark Arbeter, chief technical strategist at S&P.  He explained that he had developed a system to predict market corrections.  He looks for Nasdaq volume 40% higher than the NYSE over a three-week period.  It has provided signals in 2004, 2005, and 2006.

The second guest was Paul Hickey, an analyst at Birinyi Associates.  While acknowledging that there was always the chance of a correction, he put the odds at no greater or worse than at any other time.  Hickey noted that conclusions about the current bull market were often erroneous because the increase in stock prices has been far more gentle than in past cycles.

Erin gave Arbeter a chance to reply.  He acknowledged that he was bullish on the year as a whole, but thought there were be a better opportunity to buy in March or April.

So far so good, but now Erin slipped in one last question to make things more interesting.  She asked if Mark saw a chance, not of a correction of eight percent or so, but the possibility of a crash like that of 1987 (where there was a one-day decline of 23%).  She asked this question (we suppose) partly because experts on corrections might know about crashes.  It might also have been because Jim Paulsen of Wells Capital Management, a frequent CNBC guest, has recently been suggesting this possibility. [link corrected 2/7  JM]

Two Bad Things Happen

The first bad thing to happen was that Mark Arbeter answered the question without offering any warning to the viewer.  He responded that if the market did not correct as he was predicting, it might lead to something worse later in the year.

The problem is that Mark Arbeter had exceeded his expert status, something we have warned about.  Even if one accepts his system for predicting corrections (more on that in a minute), he has no evidence that the lack of a correction will lead to a crash.  He should have said, "That is Not My Job."  It must be difficult to say that when Erin asks one more question.

Anyone following Arbeter's advice must first wait out his prediction --70% chance of a correction by April.  Then, if he was wrong, the investor should be even more afraid of a big decline.  And this is from an analyst who says he is bullish on the year!

The second bad thing is the result of the new CNBC web site.  We are enjoying the ability to go back and replay video segments that we missed.  It is also handy to have a summary of the segment in text form, a speedier way to get the message.  If the story looks interesting, the video is there for more detail.  The problem is that the text segment (at least at the time of this writing) has a big mistake.  It reports as follows:

U.S. Treasury Secretary Henry Paulson has said that he’s expecting a crash on the scale seen in October 1987. Arbeter does see potential for that to happen, which is why he’s hoping the market corrects now rather than building the entire year and then making a major adjustment in the third or fourth quarters.

“I would be worried if we didn’t get the correction now, early in the year,” Arbeter says.

This had us checking the video, to see if we had missed some big story.  Erin clearly says "Jim Paulsen" not "Hank Paulson".  If the former Goldman CEO and current Treasury Secretary predicted a market crash, it would be big news.  There is no obvious way to notify CNBC of this error, or we would do so.  Meanwhile, anyone reading the summary could make a big financial mistake.

And Finally, the Debate

It is difficult to evaluate a system for predicting corrections without looking at the data, but Arbeter's system sounds a number of warning bells.

• He describes his three recent accurate forecasts.  We like to see a much longer time period.  Surely there are some "false positives" as well as accurate calls if one looks further back.  The on-screen graphic said "seventy percent."
• How has he tested the parameters?  For example, what is defined as a correction?  How did he arrive at 40% as the Nasdaq/NYSE volume difference.  What if it were only 37%?
• How does he determine the length of time before a correction will occur?

Our experience with such systems is that by playing around with the values for a few variables it is easily possible to find something that seems to explain some past corrections.  A system developed in that way has questionable predictive power.

One very obvious point is that tech stocks have lagged the market during the current rally.  Would it be so surprising to see those sectors catch up for a bit?  Wouldn't Nasdaq volume gain if that happened?  What if the Vista launch does stimulate a new equipment buying cycle?

By contrast, we find Paul Hickey's conclusions to be quite sound.  There is certainly a chance for a correction, but our research shows that market bottoms are easier to identify than tops.

In particular, we recommend that our readers pay close attention to Hickey's careful job of distinguishing this bull market from past cycles, something we have also tried to demonstrate.

Conclusion

How stories are communicated may prove to be just as important as the story itself.  MSM web sites do not offer the same potential for instant correction as there is for an investment blog.  Maybe the medium is the message.

Typical Research Mistakes

A noted quantitative analyst from a major firm made a series of predictions over the course of some months using the decline in the ISM survey as a sign of economic collapse.  Taking note of some methodological problems with the work, I watched with interest.  What would happen when the ISM survey showed an uptick?  Would the analyst change his predictions?

What actually happened is typical of Wall Street research.  While I have carefully studied all of the reports, I cannot cite chapter and verse from this proprietary research.  Besides, it could be almost anyone guilty of these two serious errors.

First, I was not surprised to see that his predictions remained the same.  He just changed the model so that it would make the same forecast!  His readers now were to use the LEI as the true leading indicator not the ISM.  Many quantitative analysts "refine" their models to fit what has happened in the last couple of years.  This frequent backfitting of models seems to make them look smart, whether they have been right or wrong.  But it does not work.  Models should help to identify deviant situations, not constantly change to fit the mood of the moment.  Genuine backtesting would reveal the weakness of this approach.

Second, there is a major problem in using the LEI series for techniques like multiple regression. Those watching last week's release of the LEI may have noted that the Conference Board announced a small miss for October while making a significant upward revision for September.  The Conference Board maintains that these indicators have a true "leading" quality.  Maybe so, maybe not.  Anyone developing a regression  model should use the numbers first reported, not the eventual revisions.  Since the researchers did not state that they did so, I suspect that they just used the posted historic series.

The LEI uses ten different indicators.  Not all of these indicators are in final form when the first report comes out.  So be it.  The Conference Board is doing its best, just as the BLS and other government agencies do their best in reporting and revising data.

We do not always have the data we need at the time we want.  Pretending otherwise just leads us astray.  Our team has found that the most common error in model-building is a failure to specify correctly what data were actually available at the time the forecast must be made.