When Can You Trust a Forecaster?

Summary

• A Federal Reserve Bank of Cleveland working paper documents systematic differences in forecasting accuracy among participants of the ECB’s SPF.
• Forecaster performance varies over time and across macroeconomic environments, with accuracy lowest during the GFC.
• Participants’ forecasting ability exhibits clear patterns, with individuals either consistently outperforming or underperforming relative to the average.

Introduction

Individuals participating in forecaster surveys typically make mistakes. However, central bank survey makers assume these mistakes fall away relative to other participants over time. That is, no professional forecaster is systematically better or worse at predicting the future than other participants. But a new Federal Reserve Bank of Cleveland working paper shows that this is untrue for the European Central Bank’s (ECB) Survey of Professional Forecasters (SFP).

Specifically, the authors find that particular forecasters are consistently better at forecasting during more volatile economic periods, while others display higher relative accuracy in more tranquil environments. There is also evidence that certain participants outperform in all economic conditions and so could have higher innate ability – though this is not the paper’s main focus. The paper also shows how average forecaster performance varies over time.

The results suggest that central banks must consider persistent heterogeneity in participants’ forecasting performance. The current ‘expectations formation process’ on which central banks found their survey models does not allow for these systematic differences, however, so the surveys are unlikely to elicit the most accurate forecasts from respondents.

Expectations Formation Process

The full-information rational expectations (FIRE) model assumes that all agents know the economy’s true structure and can access the same information set. It implies that all agents are identical and therefore cannot generate the type of dispersion in agents’ expectations – that is, disagreement – observed in surveys or financial markets.

FIRE was recently replaced with a weaker form of rational expectations in which informational rigidities (IR) generate disagreement at the aggregate level. Missing, however, is the possibility that individuals differ systematically in their forecasts over time. That is, agents can display differences in their forecasting behaviour at a point in time, but under IR their observed behaviour over time should, on average, be the same.

The paper’s central focus is whether this is true. Do participants display comparable forecast accuracy, or are there systematic differences? And if so, what are the sources of such differences?

Data & Methodology

The authors use point and density forecasts from participants in the ECB’s SPF. The survey produces euro-area expectations for real GDP growth, inflation as measured by the harmonized index of consumer prices (HICP), and the unemployment rate. They evaluate the rolling one-year-ahead and one-year/one-year-forward forecasts from surveys conducted between 1999 and 2018.

They include only participants for whom they have at least 50 forecasts over the 20-year period for real GDP, unemployment, and inflation. The authors measure the point forecasts’ accuracy as the absolute error between the actual (including revisions) and predicted value. For the density forecasts, they adopt an absolute rank probability score (ARPS), which credits a participant when they assign probability in bins close to the bins containing the actual outcome.

To assess forecasting accuracy across individuals, the authors compare a participant’s performance relative to the average, controlling for participant-fixed effects (α) and a time-varying (λ) component. If the participants are all broadly equal over time (α=0, λ=1), their performance matches that of the average. Participants may, however, be systematically better (α<0, λ<1) or worse (α>0, λ>1) than average. Alternatively, performance may vary by forecasting environments, where participants are relatively more accurate in a tranquil (α<0, λ>1) or volatile environment (α>0, λ<1).

Results

The authors generate results for around 30 participants across three variables (output, inflation and unemployment) and two sets of predictions (point and density forecasts). On aggregate, forecast performance varies significantly over time, with errors highest during the Global Financial Crisis (GFC) (Chart 1; the y-axis represents forecast accuracy based on absolute error or ARPS, and a higher value indicates lower accuracy).

Furthermore, average performance is higher when the accuracy of one of the three variables is higher. For example, a better one-year ahead GDP forecast often comes with better inflation and unemployment forecasts. That said, the link between GDP and inflation is weaker than GDP and unemployment. One-year-ahead average forecast performance also improves with one-year/one-year-forward accuracy.

Comparing individual performance relative to the aggregate, the authors find significant systematic differences. With their density forecasts, participants consistently outperform or underperform relative to the average across all economic environments. That is, they more frequently assign probabilities to future outcomes that are on average more or less accurate over time.

Point estimates are generally less accurate. Also, contrasting density forecasts, participants’ relative predictive performance varies with the forecasting environment. That is, some participants perform better in high-variance periods but worse in low-variance periods (Chart 2). It is more typical to observe individuals who become more accurate in high-volatility periods than vice versa.

Chart 2’s blue line tracks participants who consistently underperform relative to the average. The purple line shows individuals who are consistently more accurate in all environments relative to the average. The orange line shows participants who are better at forecasting in low-volatility environments, while the red those who improve in high-volatility environments and sometimes outperform the most accurate individuals.

Bottom Line

By assuming individual forecasters have on average similar forecasting abilities over time, central banks incorrectly elicit expectations of future growth, inflation and unemployment. They could increase accuracy by accounting for differential effects of the forecast environment on predictive performance. More broadly, forecaster performance is generally worse for output and during periods of high volatility.

Citation

Rich, Robert, and Joseph Tracy, (2021), “All Forecasters Are Not the Same: Time-Varying Predictive Ability across Forecast Environments.” Federal Reserve Bank of Cleveland, Working Paper (21-06). https://doi.org/10.26509/frbc-wp-202106.