Experience shows that what happens is always the thing against
which one has not made provision in advance.
— John Maynard Keynes1
Our best plan is to plan for constant change and the potential for instability, and to recognize that the threats will constantly be changing in ways we cannot predict or fully understand.
— Timothy Geithner2
The economics profession has been appropriately criticized for its failure to forecast the large fall in U.S. house prices and the subsequent propagation first into an unprecedented financial crisis and then into the Great Recession. In this post, I examine the performance of the forecasts produced by the economic research staff of the Federal Reserve Bank of New York (New York Fed) over the period 2007-10 and consider some of the reasons why we, like most private sector forecasters, failed to predict the Great Recession. This spreadsheet contains staff forecasts, the outcomes, and a standard measure of private sector forecasts—the Blue Chip consensus. In addition, staff material prepared for bi-annual meetings of the New York Fed Economic Advisory Panel provide some further insights into the evolution of the staff outlook.
The staff forecasts of real activity (unemployment and real GDP growth) for 2008-09 had unusually large forecast errors relative to the forecasts’ historical performance, while the forecasts for inflation were in line with past performance. Moreover, although the risks to the staff outlook were to the downside throughout this period, it wasn’t until fall 2008 that a recession as deep as the Great Recession was given more than 15 percent weight in the staff assessment.
How Bad Were the Forecasts for Real Activity?
Economic forecasters never expect to predict precisely. One way of measuring the accuracy of their forecasts is against previous forecast errors. When judged by forecast error performance metrics from the macroeconomic quiescent period that many economists have labeled the Great Moderation, the New York Fed research staff forecasts, as well as most private sector forecasts for real activity before the Great Recession, look unusually far off the mark.
One source for such metrics is a paper by Reifschneider and Tulip (2007). They analyzed the forecast error performance of a range of public and private forecasters over 1986 to 2006 (that is, roughly the period that most economists associate with the Great Moderation in the United States).
On the basis of their analysis, one could have expected that an October 2007 forecast of real GDP growth for 2008 would be within 1.3 percentage points of the actual outcome 70 percent of the time. The New York Fed staff forecast at that time was for growth of 2.6 percent in 2008. Based on the forecast of 2.6 percent and the size of forecast errors over the Great Moderation period, one would have expected that 70 percent of the time, actual growth would be within the 1.3 to 3.9 percent range. The current estimate of actual growth in 2008 is -3.3 percent, indicating that our forecast was off by 5.9 percentage points.
Using a similar approach to Reifschneider and Tulip but including forecast errors for 2007, one would have expected that 70 percent of the time the unemployment rate in the fourth quarter of 2009 should have been within 0.7 percentage point of a forecast made in April 2008. The actual forecast error was 4.4 percentage points, equivalent to an unexpected increase of over 6 million in the number of unemployed workers. Under the erroneous assumption that the 70 percent projection error band was based on a normal distribution, this would have been a 6 standard deviation error, a very unlikely occurrence indeed.
Did We Calibrate the Risks to the Forecast Appropriately?
Of course, there is much more to forecasting than the point forecasts reported in the spreadsheet. In particular, it is crucial to assess the uncertainty and risks around any point forecast.
Throughout this period, the uncertainty and downside risks assessed around our point forecast were substantial relative to economic fluctuations in the Great Moderation. One way of gauging the appropriate calibration of downside risk is to measure the depth of the implied recession if the risks were realized.
The chart below does this by considering the probability distribution of the four consecutive quarters with the lowest GDP growth in a recession. It presents results based on the staff outlook in April 2008 and November 2008. The depth of the mild recessions shown in the chart was typical of the type of recessions expected during the Great Moderation. The actual depth of the 2007-09 recession as gauged by this metric is currently estimated to be 5 percent. As the chart shows, it was only by November 2008 that the probability of the actual outcome was above 15 percent.
Upon seeing this type of calculation, Robert Barro, a Harvard professor and member of the New York Fed Economic Advisory Panel, noted that the decline in real stock market values in the United States was similar to that observed in countries experiencing depressions. Taking this relationship into account in the calibration of the downside risks produced about a 50/50 chance of the currently observed depth of the Great Recession (see the March 2009 Wall Street Journal op-ed article by Barro for his assessment of the probability of a depression).
Recent Forecast Performance
In contrast, the New York Fed staff forecasts for 2010 made in 2009 and early 2010 are quite accurate (under the assumption of no major revisions to the estimates of GDP growth in 2010). This accuracy, however, has not been sustained through 2011. As widely discussed by a number of Federal Reserve officials, the level of real activity in 2011 has been disappointing relative to expectations. This shortfall is evident in the chart below, which compares forecasts for GDP growth in 2011 and 2012 produced in April and October 2011. However, this chart also depicts the uncertainty and risks around the staff forecast as of April 2011. Given the uncertainty around the April forecast, the subsequent changes to the outlook are not very surprising. On the other hand, near-term downside risks to this forecast were low compared to other forecasts produced in the last four years, so the direction of the change was more surprising.
Why Did We Fail to Forecast the Great Recession?
The quotations from Keynes and Geithner at the start of this post capture the importance of constantly striving to ensure that policy is robust to unexpected events. As explained in much of the recent work of the 2011 Nobel Prize–winning economist Tom Sargent, the unexpected events for which policymakers need to make provision have the characteristic of being the most likely unlikely bad event. The collapse in housing prices and its propagation to the economy certainly fit this description.
A leading example of how effective a robust approach to policymaking can be is the 2009 Supervisory Capital Assessment Program. In this program, large U.S. banks were evaluated against a capital standard under the assumption of a longer and deeper recession than contemplated in the prevailing consensus estimate. The idea was that if banks had sufficient capital to continue performing their intermediation function under this more adverse scenario, the scenario was less likely to occur.
Indeed, in the period leading up to the financial crisis, analysts who were suspicious of the stability of the Great Moderation, such as Nouriel Roubini, offered assessments that proved to be significantly more accurate than the point forecasts of New York Fed research staff or most professional forecasters in gauging the potential for unlikely bad outcomes. On a more positive note, if one compares the downside risk in the New York Fed research staff outlook with that of the Survey of Professional Forecasters (see the chart below from April 2008), there is some evidence that we had a more sober assessment of the risk of a severe downturn than did private sector forecasters.
Looking through our briefing materials and other sources such as New York Fed staff reports reveals that the Bank’s economic research staff, like most other economists, were behind the curve as the financial crisis developed, even though many of our economists made important contributions to the understanding of the crisis. Three main failures in our real-time forecasting stand out:
- Misunderstanding of the housing boom. Staff analysis of the increase in house prices did not find convincing evidence of overvaluation (see, for example, McCarthy and Peach [2004] and Himmelberg, Mayer, and Sinai [2005]). Thus, we downplayed the risk of a substantial fall in house prices. A robust approach would have put the bar much lower than convincing evidence.
- A lack of analysis of the rapid growth of new forms of mortgage finance. Here the reliance on the assumption of efficient markets appears to have dulled our awareness of many of the risks building in financial markets in 2005-07. However, a March 2008 New York Fed staff report by Ashcraft and Schuermann provided a detailed analysis of how incentives were misaligned throughout the securitization process of subprime mortgages—meaning that the market was not functioning efficiently.
- Insufficient weight given to the powerful adverse feedback loops between the financial system and the real economy. Despite a good understanding of the risk of a financial crisis from mid-2007 onward, we were unable to fully connect the dots to real activity until 2008. Eventually, by building on the insights of Adrian and Shin (2008), we gained a better grasp of the power of these feedback loops.
However, the biggest failure was the complacency resulting from the apparent ease of maintaining financial and economic stability during the Great Moderation. Perhaps most important, as noted by some analysts as early as the 1990s, these adverse consequences of the Great Moderation were most likely to arise from the actions, judgments, and decisions of financial market participants:
Longer stretches of economic growth imply greater leverage and complacency and thus, greater financial problems when recessions do occur.
–William Dudley and Edward McKelvey3
1Letter to Jacob Viner, June 9, 1943, Collected Writings of John Maynard Keynes, ed. Donald Moggridge, vol. 25. London: Macmillan, 1980.
2Letter from the Chair, Financial Stability Oversight Council Annual Report.
3The Brave New Business Cycle: No Recession in Sight, Goldman Sachs Economic Research Group, January 1997.
Disclaimer
The views expressed in this blog are those of the author and do not necessarily reflect the position of the Federal Reserve Bank of New York, or the Federal Reserve System. Any errors or omissions are the responsibility of the author.
RSS Feed
Follow Liberty Street Economics
Potter’s essay identifies 3 oversights causing the forecast failures. It is of course important to learn from our mistakes, and let’s be optimistic and assume that these particular oversights will not be “oversighted” next time around. But in the spirit of the quotes from Keynes and Geithner, the trick is not only to learn from mistakes, but to learn how to deal with uncertainty, ignorance and surprise. Economies are so complicated and dynamic that learning from our errors does not mean that we will ever exhaust the inventory of surprises. I don’t have all the answers on how to model and manage Knightian uncertainty. I agree that concepts of robustness are central. I would like to draw readers’ attention to my most recent book: Yakov Ben-Haim, 2010, Info-Gap Economics: An Operational Introduction, Palgrave-Macmillan. See descriptions at: http://info-gap.com/content.php?id=85 I present a distinctive approach to robust policy formulation that is operationally distinct from the min-max methods develop by Sargent and many others. There are no panaceas, but the info-gap approach has been applied in many areas and has something to offer to economics as well. For those who prefer a less technical approach you are welcome to visit my blog on “Decisions and Info-Gaps” at: http://decisions-and-info-gaps.blogspot.com An explicitly economic post is “Doing Our Best: Economics and Optimization” at: http://decisions-and-info-gaps.blogspot.com/2011/08/doing-our-best-economics-and.html
Economists – the only section in society to have made no productivity gains in 50 years!
The New York Fed deserves some credit for posting articles for comment, reading the comments and responding to them in a thoughtful manner. This is something sadly lacking in the White House, Treasury and the Congressional Budget Office.
Unirealist makes a number of claims in his post. I would suggest that looking through FOMC transcripts (available through the end of 2005) gives a very different impression. In particular, the June 2005 transcript contains a detailed discussion of the housing situation where Chairman Greenspan directly challenges the modeling assumption used in the house price work by McCarthy and Peach. Robert Sherman indirectly makes the argument of the blog for me. Of course there were big financial shocks before the Great Recession but in the Great Moderation one could not see their imprint in real activity data. Joe Brennan asks about the composition of the Blue Chip Consensus forecast. Usually over 50 business economists, some but by no means all of which do work for the sell side, contribute forecasts each month. The consensus is the average of these forecasts. A number of statistical studies have shown that this average of forecasters is at least as accurate as any individual forecaster when accuracy is measured over a long period of time. Of course, this does not mean that the Blue Chip consensus is always the best forecast, the failure to forecast the Great Recession being the best example.
There were plenty of economists who timely predicting both the bust of the housing bubble and the consequence being a recession; some can be found here http://www.investorhome.com/predicted.htm With the bust of the first international housing bubble the probability of a global recession was close to 100pct.
The wealthiest people in this world count more high-school dropouts among their ranks than economists. If the market is the determinant in matters economic, it speaks clearly that economists at their trade are worse than high-school dropouts in determining which way the economy is going.
Although I don’t agree with various aspects of it, I found the post to be interesting. It should also be noted that many qualitative economic interpretations and commentary of the period proved quite inaccurate. This morning, I wrote a response to this blog post. Here is an excerpt concerning an aspect I believe to be of paramount importance: “The inability to predict “The Great Recession” should serve to cast uncertainty on forecasters’ ability to predict future severe economic weakness, especially since the level of complexity inherent in the overall economic environment is, according to my analyses, growing.” http://economicgreenfield.blogspot.com/2011/11/economic-forecast-efficacy-of-recent.html
You need to work through Steve Keen’s lectures here: http://www.youtube.com/watch?v=KWUG1n1jEJI&feature=list_related&playnext=1&list=SP0A21A329D01D0CFE This gives the analytical basis for understanding macroeconomics – something that Geitner and Bernanke clearly do not grasp.
“The Great Moderation” from 1986 to 1996? Are you serious? Did you not read about the 1987 Crash, the Dot Com Boom, Long Term Capital Management? The only ones who did not see the recession coming were the economists and policy makers at the NY and DC Fed offices and their counterparts in Treasury and the now inept Congressional Budget Office. All the policy makers did was put off a real recession by replacing one bubble with another. Those of us economists in the business world who actually think through developments were well aware that real estate prices were too good to be true. Crash! They were. But the policy makers have kept the air in the bubble by injecting one false start after another instead of letting the market settle. What a horrible disservice. Try some rational thinking out for a change. Perhaps you’ll understand then why many of us prefer flat rates and an end to crony capitalism and state sponsored socialism.
2011 is finally history. The economic recovery, which officially began in 2009, was scarcely evident as the US economy muddled through 2010. It seemed that for every piece of good news, like the strong end to the 2011 Christmas shopping season, was countered by news of a setback, such as unemployment rates that unexpectedly returned to nearly 10% during the same period.
(con’t) And, I might add, if the Fed does not bother to honor the terms of its dual mandate, which states in uncertain language that prices are to be stable. A 2% price inflation rate, whether targeted or not, isn’t “stable.” Stable would be a 0% rate. What do we care that a forecast “failed” if the basis for the forecast was a violation of the statutory mandate in the first place?
It seems to me–and no doubt to many others in anti-Fed land–that the appeal to “failed forecasts” is disingenuous. It seems, in fact, that the Fed, particularly under the questionable guidance of Alan Greenspan, operated under a politically-motivated agenda that had the deliberate intention of blowing a housing bubble, and “forecasts” were irrelevant. Now that he is gone, the Fed could do a great service to the country with transparency about his reign there. Is it true that Greenspan visited the WH on a regular basis after 9/11, as often as 2-3 times a week? Why would he confer with with POTUS and his advisors in such unprecedented frequency unless it was about blowing a financial bubble for political reasons? And Greenspan was not an unintelligent man, no matter what else can be said about him. He had to have known that a housing crash would be the ultimate consequence. It doesn’t take a large measure of cynicism to suppose that he had a pretty fair idea that the bubble would last through Bush’s terms in office, and burst well after he himself retired from the helm. And if all that is true, how much naivite would it take to suppose that Bernanke took over the chairmanship blind to what was going on? So let’s not get caught up in finger-pointing about forecasts. They really aren’t important. At least, not if the Fed is willing to act as a political arm of the Presidency.
No question that the Fed intentionally misled the sheeple, at the behest of the big banks, in order to support the “sell side consensus”. It has been too easy to revise forcasts and results, after the fact, with the press willing to look the other way and give little coverage of the revisions.
MODEL MASTURBATATION Thank you for posting my unedited comments on your blog. I have to thank Janet Tavakoli for coining the above title phase and adding it to the economic lexicon. As an aside I note your use of the phase ‘Blue Chip consensus’ and it’s tight correlation to the New York Fed Forecast. Does this blue chip consensus refer to what is commonly called, ‘Wall Street Consensus’, ie ‘sell side consensus’. I rest my case.
(Continued) Lee Adler notes that New York Fed staff were not alone in making these forecasting errors. Indeed, one of the sheets in the Excel workbook attached to the post shows the tight correlation between the New York Fed staff forecast and the Blue Chip consensus. This tight correlation would be present for the other forecasters Mr. Adler lists. The spreadsheet also shows that GDP growth for 2008 has been revised over the last three years from mildly negative to -3.3 percent. Much of this decline is estimated to have occurred in the fourth quarter of 2008. If forecasts were evaluated against the initial estimate of GDP, they would not look as bad. Of course, the unemployment rate is not revised (small changes do occur, but only because of new seasonal factors), so this is perhaps the best way to measure the accuracy of a forecast for real activity.
Many commenters to the blog have noticed that their comments do not necessarily appear immediately. As described in our comment guidelines: “Comments are moderated and will not appear until they have been reviewed to ensure that they are substantive and clearly related to the topic of the post.” In addition, we also screen the comments for vulgarity, threats, etc. For example, Joe Brennan submitted a comment that was substantive and directly related to the topic of the post. However, he used an analogy that required us to think hard about whether to post the comment. In the end, we decided to post the comment, but it would have been easier to post quickly if he had commented, “be careful about falling in love with the predictions of your models,” which in my opinion was one of the main conclusions of his comment and the blog post itself. We are relaxing some of the other restrictions on comments, notably, the twenty-four-hour comment period. David Andolfatto correctly notes that predicting the exact timing of a recession is difficult, but I disagree with his claim about conditional forecasts. Very few economists conditioning on a recession were predicting the depth of the recession that occurred; there is a section of my post discussing this point (here, the population of economists I am applying “few” to is Ph.D. economists working for government agencies and large banks, an important caveat many of the commenters have pointed out). Dr. Andolfatto uses the notion of a crisis to avoid this implication about the depth of the recession. I agree that we should condition on a crisis and think both about how it might unfold and, perhaps more importantly, how conditions might develop that make a crisis more likely.
Mr. Potter- I realize that you staff guys are not the real villains here. Sorry I spewed at you. Thank you for putting my comment back. However, I also want to point out that not only did NY Fed staff get the forecast consistently wrong, so did every single FOMC member and every single district bank president. Not only did all of them get the forecasts wrong every time, they even could not accurately say where the economy was at the time they were surveyed. This, to me is evidence of ingrained delusional crowd behavior. http://wallstreetexaminer.com/2010/07/26/the-fed-clueless-delusional-or-both/ To their credit, while they could not get the GDP or employment figures correct, they did get inflation largely correct. Congress should return the Fed to the single mandate. The idea that the Fed should additionally target GDP, is beyond laughable. It’s bizzaro world stuff. But there are some who take it seriously.
These reflections on failure are reminiscent of ‘I didn’t know the gun was loaded’, in short, another excuse at 6pm on a Friday. We hope no one notices. You may need that excuse book again shortly – looks like there’s a better than 80% probability of another recession, if we’re not there already. Do your current model inputs compute a recession? You’d better start model masturbating again. Suggest you subscribe to John P. Hussman’s Weekly Market Comment and save some money on your model drones.
Nobody saw it coming; Umm, except the entire Austrian school of Economics, shadowstats, anybody that uses honest metrics, and uhhh this guy back in 2001 http://www.youtube.com/watch?v=kFd8YluIVG4
George Felis points out some of the possible sources for poor forecast performance. The theme of my blog post was that the apparent tranquility of the Great Moderation lulled many forecasters into a false sense of security. So in the three categories that Felis put forward, our forecast and those of many private sector forecasters exhibited statistical bias. The quotes from Keynes and Geithner capture one approach to dealing with this issue; the quote from Tolstoy used at the start of Michael Lewis’ The Big Short: “The most difficult subjects can be explained to the most slow-witted man if he has not formed any idea of them already; but the simplest thing cannot be made clear to the most intelligent man if he is firmly persuaded that he knows already, without a shadow of doubt, what is laid before him.” captures well Felis’ historical viewpoint. Clearly, there were some forecasters who viewed the Great Moderation with suspicion, such as Roubini, and provided more accurate forecasts. It is also true, as Lee Adler points out, that many who predicted a bust in the housing market were viewed as Cassandras. Part of the reason for this is that some of these forecasters did not accurately predict the timing of the fall in house prices. One lesson we have taken on is to listen more carefully to views that run against the conventional wisdom even if their short-run forecasts do not pan out. GeorgeK is correct to emphasize the role of innovations in mortgage finance leading up to the crisis. In the post I linked to the work of Ashcraft and Schuermann that details the misaligned incentives in the mortgage origination chain. The growth of “shadow banking” documented in a staff report (http://www.newyorkfed.org/research/staff_reports/sr458.pdf) from the New York Fed took place outside of the traditional regulatory structure. This is another lesson we have taken from the crisis–an understanding of all parts of the financial system, not just the regulated core, is crucial.
“It’s almost worth the Great Depression to learn how little our big men know.” – Will Rogers
Simon, The economics profession has been appropriately criticized for its failure to forecast the large fall in U.S. house prices and the subsequent propagation first into an unprecedented financial crisis and then into the Great Recession. As a member of the economics profession, I take issue with you lumping me into a group of people that(according to someone’s infallible judgement) are to be “appropriately criticized” for their apparent lack of skill in the divine art of haruspicy. You are giving people the impression that economists should be able to forecast business cycle turning points and asset price movements with precision. All we can really do is predict that crises will occur; and then make conditional forecasts about how things are likely to evolve after that. We know that the people of Naples are one day going to suffer from the wrath of Vesuvius. Everyone knows this (but choose to ignore it, because the probability is small). But when that mountain blows, or when the tsunami hits the coast of California, you can bet there will be people complaining that the “authorities failed to predict…and that therefore I am entitled to…)
You failed to take into account that bankers were/are involved in a criminal enterprise, looting their banks. The banks pushed the toxic mortgage origination simply for more product to securitized which generated fees and obscene bonuses. When they ran out of mortgages they simply created CDO’s to keep the up the scam. Government regulators at all levels failed to do their jobs including a warning from the FBI about criminal activity in the mortgage markets. The problems is not your models; the problem is regulatory capture; which is why OWS has struck a cord with what’s left of the middle-class.
When a prediction does not match reality, there are three possibilities. The behavior is too complex to predict, the predictor is incompetent, or the predictor is biased. You seem to think “Nobody saw it” is a good excuse, when in fact a large number of people saw the upcoming collapse, warned about it, and were roundly bashed at the time, and ignored afterwards. But this is not a new phenomena, a student of history should be well aware of the phrase “you have hidden these things from the wise and learned, and revealed them to little children” Myself, I would rather believe bias.
Running count of posters from another site = hundreds and you don’t have any up and no way to follow them. You are crooked bankster slime. Let the world know what others think for once instead of deleting comments. You only prove you’re worthless and incapacity of rational thought.
Really? You can’t keep a comment up and no way to read other comments? Everyone must LOVE you guys. Please note, you suck! You think that your machinations are saving the world and yet you are the very reason it is in this condition. The world will not be safe or free until there is no more FED and no more bankers. I only hope I live to see the day.
Alan Greenspan thwarted regulating the mortgage industry at the same time the Fed was suppressing interest rates and now you tell the world (at 6PM EST on Friday after Thanksgiving)that the wizards at the FED had a “Misunderstanding of the housing boom”. Didn’t Fed Chairman Greenspan encourage borrowers to take out adjustable rate mortgages shortly before the housing markets started to implode? Wasn’t Chairman Bernanke for all of 2006 assuring the world that the imploding mortgage backed security market would not affect the general economy? Let us review: super easy money from the Fed, lax or no regulation of criminal mortgage practices, and a lack of understanding how the economy works — why is the Fed in control of the financial system?
No suprise Bernake and the Fed haven t gotten a single thing correct in 6 years. Before that it was Greenspan, a man who will eventually be known as a modern day John Law. USD is doomed, the FED an abject complete failure. Short em to zero…wooooza
The clowns in academia fail again. What s new. The financial engineers and our political class in an ongoing incestuous relationship. This will end badly, but not until Charles Ponzi and his merry band of fed lackies are out of business. Whats the next wall street scam? Leveraged buyouts, worthless internets, MBS. How about junior miners….i ll bite on that one. Short em to zero. So says I , Merciless.
The excuse that most other professional forecasters didn’t foresee it is just that, an excuse. Some professional forecasters did see it. They were derided as Cassandras and dismissed by Wall Street and Fed insiders, who are only beholden to each other, and to their own delusions. Millions of amateur economic forecasters who frequented the financial message boards and blogs saw what was happening and what was coming. They had one important advantage. They live in the real world, not inside the Beltway, not within the marble halls and equally hardened thought processes of the Fed, and not in the ivory towers of academia, a word which sounds like a disease, because it is a disease. Not only do these environments cause delusional thinking, they attract delusional people. The same is true of policy makers. I call it elitist personality disorder. It leads to delusions of grandeur, delusions of omniscience and omnipotence, and the unwillingness to take responsibility for failure and incompetence, instead engaging in blame shifting.