What Does Disagreement Tell Us about Uncertainty? - Liberty Street Economics
Liberty Street Economics

« Worker Flows in Banking Regulation | Main | Historical Echoes: Metaphors for Monetary Policy and Some GMAT Nostalgia »

January 07, 2015

What Does Disagreement Tell Us about Uncertainty?



Blog-Disagreement-iStock_000017906875Large_450

Uncertainty is of considerable interest for understanding the behavior of individuals as well as the movements in key macroeconomic and financial variables. Despite its importance, direct measures of uncertainty aren’t widely available. Because of this data limitation, a common practice is to use survey-based measures of forecast dispersion—reflecting disagreement among respondents—to proxy for uncertainty. Is this a reliable practice? Here, we review the distinction between disagreement and uncertainty as concepts, and show that this conceptual distinction carries over to their empirical counterparts, suggesting that disagreement is not generally a good proxy for uncertainty.

        The terms “disagreement” and “uncertainty” refer to very different concepts. Disagreement refers to a collection of forecasts or point predictions and the nature of their clustering around each other—the more disperse the forecasts, the greater the extent of disagreement among the survey respondents. On the other hand, uncertainty refers to the distribution of the probabilities that a respondent attaches to the different possible outcomes of the forecasted variable—the more confidence held by a respondent, the tighter this distribution is and the lower the respondent’s uncertainty.

        Although disagreement and uncertainty are different concepts, some commentators have drawn a connection between the two. That is, episodes characterized by high (low) disagreement are viewed as indicative of high (low) uncertainty shared by the respondents. This assumption provides the basis to use disagreement as a proxy for uncertainty when measures of the latter magnitude aren’t available.

        Is the assumed positive association between disagreement and uncertainty plausible? Yes. Is it necessarily true? No. To understand why either case is possible, we can look at the following figure previously discussed in this paper by Victor Zarnowitz and Louis Lambros:


Zarnowitz Lambrose


        The illustration on the left shows the forecasts and associated probability distributions of two hypothetical survey respondents—respondent A and respondent B. The close proximity of the forecasts (ŷA and ŷB) indicates low disagreement, while the tight distributions around each forecast indicate low uncertainty. However, the illustration on the right depicts another possible situation. Here the forecasts are unchanged, so disagreement remains low. But the probability distributions around each forecast are now much wider, indicating high uncertainty. The figure is important for two reasons. First, it bears directly on the question of the reliability of disagreement as a proxy for uncertainty. While the conditions depicted in the left illustration might justify this practice, the conditions depicted in the right illustration would not. Second, it shows that the dispersion of forecasts by itself is not necessarily informative about the level of uncertainty across respondents. Because most surveys only report the respondents’ forecasts, observing the degree to which the forecasts cluster together can’t tell you whether the illustration on the left or on the right in the figure is the more relevant situation.

        So, is it possible to say something more definitive about which of the two illustrations may be a better description of the relationship between disagreement and uncertainty? The answer is yes, because there are a few surveys that have the unique feature of asking respondents to report both a forecast and their uncertainty—the latter being represented by a histogram in which respondents assign probabilities to pre-specified bins that provide a range of possible outcomes for the forecasted variable. The best-known of these surveys are the U.S. Survey of Professional Forecasters (US-SPF), the European Central Bank Survey of Professional Forecasters (ECB-SPF), and the Bank of England Survey of External Forecasters (BOE-SEF).

        There are various techniques, such as calculating the variance/standard deviation or interquartile range, that can be used with these histograms to derive an estimate of each respondent’s uncertainty, which can then be used to construct a measure of aggregate uncertainty. The degree of disagreement across the respondents’ forecasts can be measured in the same way. If disagreement is a useful proxy for uncertainty, then the two variables should display a meaningful positive association.

        As it turns out, there are several studies that have used data from the aforementioned surveys to investigate the relationship between disagreement and uncertainty. These studies include work by Boero, Smith, and Wallis (2008) for the BOE-SEF; Rich and Tracy (2010) for the US-SPF; and Rich, Song, and Tracy (2012) for the ECB-SPF. The conclusions from the analyses are quite similar—disagreement is a poor proxy for uncertainty.

        To help understand this finding, the following graph uses ECB-SPF data from the first quarter of 1999 through the fourth quarter of 2013 and plots measures of disagreement and uncertainty for one-year-ahead inflation forecasts. Details on the construction of the variables are provided in the Rich, Song, and Tracy paper cited above.


Measures of Disagreement and Uncertainty in 1-Year-Ahead Inflation Forecasts


        The average uncertainty measure shows a clear upward drift since the second quarter of 2007, denoted by the vertical line, and remains nearly 50 percent higher today than in the pre-crisis period. The disagreement measure shows a noticeable increase starting in 2008, with some fairly dramatic spikes evident during the global financial crisis. More recently, and in contrast to uncertainty, disagreement has declined and returned to levels more consistent with the range observed over most of the pre-2007 period. When we examine the series more formally, the correlation is essentially zero. If we limit the analysis to data through the second quarter of 2007 to avoid any concerns that the results may be unduly influenced by the global financial crisis, then the correlation becomes negative. There is a relatively straightforward explanation for the weak relationship—the uncertainty measure displays relatively smooth behavior while disagreement is more volatile. These features of disagreement and uncertainty depicted in the graph are very representative of the relationship for other variables, such as growth and unemployment, and other forecast horizons. In particular, while most of the other observed correlations for the ECB-SPF data were higher than zero, they were still quite low and could generally be described as weak.

        Our discussion focuses on the question posed in the title of our post: what does disagreement tell us about uncertainty? While we recognize that disagreement could be strongly and positively related to uncertainty, and that it has been common practice in many analyses to assume such a relationship, we find that there is little empirical evidence to support this view. A better approach is to use measures of uncertainty when they are available, as well as to design more surveys that would have the capability to directly measure respondents’ uncertainty.


Disclaimer
The views expressed in this post are those of the authors and do not necessarily reflect the position of the Federal Reserve Bank of New York or the Federal Reserve System. Any errors or omissions are the responsibility of the authors.





Ploenzke_Matthew
Matthew Ploenzke is a research associate in the Federal Reserve Bank of New York’s Research and Statistics Group.


Rich_robert
Robert Rich is an assistant vice president in the Research and Statistics Group.


Tracy_joseph
Joseph Tracy is an executive vice president and senior advisor to the Bank president at the Federal Reserve Bank of New York.
Posted by Blog Author at 07:00:00 AM in Education, Forecasting, Macroecon
Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Dan – We may not have fully understood your first point. Even though a person may report a single value for a point forecast, that forecast is based on an assessment of the different possible values that the target variable may take on along with an associated probability of each possible value. With regard to your second question, respondents are not instructed how to calculate their point forecast and do not necessarily provide the mean of their density (histogram) forecast. However, research by Joseph Engelberg, Charles Manski and Jared Williams in a 2009 article showed that the calculated means from the US-SPF density forecasts were consistent with the associated point forecasts. Also, in a special questionnaire fielded by the ECB-SPF, the vast majority of respondents indicated that their reported point forecast reflects the mean of their density forecast.


Chris – We’re glad you found the post interesting. We have not calculated the disagreement and uncertainty measures we describe in this blog for the US-SPF post-2002. While we can make no comment on whether the same pattern holds, there are a couple of points worth mentioning. First, without getting into technical details, disagreement and uncertainty measures from the US-SPF can only be consistently compared for the same quarter. That is, looking at the pattern for observations such as 1992Q1, 1993Q1, . . . , 2014Q1, then the pattern for observations such as 1992Q2, 1993Q2, . . . , 2014Q2 and so on. This is because the forecast horizon in the US-SPF declines from the 4th quarter of a year to the 3rd quarter of the subsequent year, resulting in both disagreement and uncertainty generally declining together during this survey cycle. So, this source of the observed co-movement between the series is an artifact of the survey design. Another point is that it will be important to consider the pre-2008 period to make sure that any correlation between disagreement and uncertainty is not unduly sensitive to the incidence of the Great Recession.

Does this hold for the US in the last decade? The sample in the study by Robert Rich and Joseph Tracy ends 2002. See figure 2, using US-SPF, from Nick Bloom's 2014 JEP paper is at least suggestive that disagreement is a helpful proxy for true uncertainty in forecasts: http://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.28.2.153. This roughly the same time frame you present using the ECB-SPF and it looks very different. Thanks for the interesting post!

Might there be a difference between a forecast for an interest rate, say, and for a report of a subjective assessment of uncertainty? A forecast of 1.0 percent, say, refers to a prediction of something observable, something that means the same thing over time and for each respondent. But can we know that subjective probabilities assigned to various bins mean the same thing over time, and even across respondents, in the same sense? 1.0% is a guess, but it's a guess about something real and observable; but a report of a 10% subjective probability of an 0.2% rate seems more of a subjective sensation, and the metric may be more "flexible" across respondents or even for a given respondent over time... no?

(A related question: are the survey respondents asked both for the points in the probability distribution and for the mean of the distribution (point estimate)? Are they forced to make probability distribution estimates consistent with their point estimates, and if not, how consistent do they turn out?)

(I'm just wondering whether we can reliably compare over time reports of subjective uncertainty the same way we can measure dispersions of point forecasts.)

The comments to this entry are closed.

About the Blog
Liberty Street Economics features insight and analysis from New York Fed economists working at the intersection of research and policy. Launched in 2011, the blog takes its name from the Bank’s headquarters at 33 Liberty Street in Manhattan’s Financial District.

The editors are Michael Fleming, Andrew Haughwout, Thomas Klitgaard, and Asani Sarkar, all economists in the Bank’s Research Group.

The views expressed are those of the authors, and do not necessarily reflect the position of the New York Fed or the Federal Reserve System.


Economic Research Tracker

Liberty Street Economics is now available on the iPhone® and iPad® and can be customized by economic research topic or economist.


Most Viewed

Last 12 Months
Useful Links
Comment Guidelines
We encourage your comments and queries on our posts and will publish them (below the post) subject to the following guidelines:
Please be brief: Comments are limited to 1500 characters.
Please be quick: Comments submitted after COB on Friday will not be published until Monday morning.
Please be aware: Comments submitted shortly before or during the FOMC blackout may not be published until after the blackout.
Please be on-topic and patient: Comments are moderated and will not appear until they have been reviewed to ensure that they are substantive and clearly related to the topic of the post. We reserve the right not to post any comment, and will not post comments that are abusive, harassing, obscene, or commercial in nature. No notice will be given regarding whether a submission will or will not be posted.‎
Disclosure Policy
The LSE editors ask authors submitting a post to the blog to confirm that they have no conflicts of interest as defined by the American Economic Association in its Disclosure Policy. If an author has sources of financial support or other interests that could be perceived as influencing the research presented in the post, we disclose that fact in a statement prepared by the author and appended to the author information at the end of the post. If the author has no such interests to disclose, no statement is provided. Note, however, that we do indicate in all cases if a data vendor or other party has a right to review a post.
Archives