Evaluation of AEO2020 and Previous Reference Case Projections
The U.S. Energy Information Administration (EIA) projects U.S. energy production, consumption, and prices each year in the Annual Energy Outlook (AEO). Every two years, EIA also produces the AEO Retrospective Review (AEO Retrospective). The AEO Retrospective compares recent history with the Reference case projections from previous editions of the AEO. It also shows the relationship between past AEO projections and actual energy indicators, and it informs discussions about the underlying models.
The projections presented in the AEO are not statements of what will happen but of what may happen given the assumptions in the underlying National Energy Modeling System (NEMS). These publically available assumptions include projections of oil prices and gross domestic product (GDP). [1] The AEO Reference case projection generally assumes current laws and regulations, and it includes current views in economic and demographic trends and technology improvements.
Although the AEO Retrospective focuses on the Reference case projections, we review the projections in the context of the AEO’s full range of cases, which illustrate the uncertainty inherent in long-term projections. Beyond the Reference case, each year’s AEO typically includes cases that vary assumptions on economic growth, oil and natural gas supply, oil price, and renewable cost. In addition, recent AEOs have also included alternative cases that examine alternative policy developments.
Methodology overview
The AEO Retrospective summarizes the relationship between the Reference case projections and realized energy usage by comparing absolute differences and the average absolute percentage differences for many of the major measures published in AEOs from 1994 to 2020. [2] The average absolute percentage difference is the simple mean of the absolute values of the percentage difference between the Reference case projections and the actual values. Most historical data are from the Monthly Energy Review (MER), which is available through the EIA open data API. [3]
Table 1 summarizes 25 comparisons based on 21 primary measures projected in the AEOs for each year. Table 2 contains the standard deviations of the percentage forecast errors (between the Reference case projections and the actual values) for every horizon H=0 to H=15 computed with AEOs from 1994 to 2020. [4, 5] For price quantities, we computed the standard deviations of the log errors. The third column of Table 1 summarizes the percentage of occurrences in which a particular measure was overestimated relative to its actual value out of the total number of annual projections for which an actual value can be compared. A zero represents a measure that was always underestimated, and 100% represents a measure that was always overestimated.
Tables 3 through 23 show the detailed results that include the Reference case projections, historical values, and percentage differences between projected and actual values for all covered years. The detailed tables also provide the average absolute difference across all AEOs for each covered year.
Previous-year statistics can change from one year’s evaluation to the next because of updates to historical data in the MER and changes in the measurement of GDP. In addition, the statistics do not account for the continual improvements made to NEMS in an attempt to more accurately capture developing energy market trends based on more recent experience.
Key findings
As reflected in the relatively lower average absolute percentage differences in the second column of Table 1, measures of energy consumption are projected with greater accuracy than measures of net imports and of energy prices. Energy consumption tends to change at a slower pace than other indicators. Many energy-using devices, such as appliances, automobiles, and industrial equipment, are expensive to purchase and have long service lives. The substantial lead time for these large purchases, along with the effects of energy contracts and other mechanisms, tend to slow changes in consumer energy demand.
For the measures with wider deviations, many factors contribute to differences between the AEO Reference case projections and realized outcomes, but two primary contributors are the initial projections of future oil prices and the overall economic activity that are used in NEMS. [6] These projections can greatly influence the other projections the model makes, which is why each AEO reviewed in this report includes alternative cases exploring differences in economic growth (Low Economic Growth case and High Economic Growth case) and in oil prices (Low Oil Price case and High Oil Price case).
In an unbiased projection, with a sufficiently large number of samples, overestimates and underestimates over time would occur in equal measure. In comparing the AEO Reference case projections with realized outcomes from 1994 to 2019, out of the 25 variables listed in Table 1, 17 more variables have historically been overestimated than underestimated.
Changes in industry-specific market conditions, significant technological breakthroughs, and new laws or regulations after the publication of an AEO can also lead to significant differences between projections and realized outcomes. For example, AEO Reference case projections from AEO1998 through AEO2017 projected the United States to continue to be a net importer of natural gas through 2017. In reality the United States became a net exporter of natural gas starting in 2017.
| Variable | Average absolute percent differences | Percent of projections overestimated |
|---|---|---|
| Gross domestic product | ||
| Real gross domestic product (average cumulative growth) [1] (Table 3) | 0.8 | 57.1 |
| Petroleum | ||
| Imported refiner acquisition cost of crude Oil (constant $) (Table 4a) | 41.7 | 28.4 |
| Imported refiner acquisition cost of crude Oil (nominal $) (Table4b) | 41.3 | 30.1 |
| Total petroleum consumption (Table 5) | 9.7 | 70.5 |
| Crude oil production (Table 6) | 17.6 | 34.1 |
| Petroleum net imports (Table 7) | 220.1 | 83.2 |
| Natural gas | ||
| Natural gas price, electric power sector (constant $) [2] (Table 8a) | 41.2 | 53.7 |
| Natural gas price, electric power sector (nominal $) [2] (Table 8b) | 43.2 | 56.7 |
| Total natural gas consumption (Table 9) | 8.7 | 58.2 |
| Natural gas production (Table 10) | 10.1 | 42.9 |
| Natural gas net imports (Table 11) [3] | 302.2 | 73.6 |
| Coal | ||
| Coal prices to electric generating plants (constant $) [4] (Table 12a) | 20.1 | 38.6 |
| Coal prices to electric generating plants (nominal $) [4] (Table 12b) | 19.4 | 41.2 |
| Total coal consumption (Table 13) | 26.3 | 78.7 |
| Coal production (Table 14) | 21.4 | 79.0 |
| Electricity | ||
| Average electricity prices (constant $) (Table 15a) | 10.2 | 29.8 |
| Average electricity prices (nominal $) (Table 15b) | 10.4 | 36.1 |
| Total electricity sales (Table 16) | 6.8 | 67.0 |
| Total energy, carbon and intensity | ||
| Total energy consumption (Table 17) | 8.5 | 79.8 |
| Delivered residential energy consumption (Table 18) | 6.8 | 71.0 |
| Delivered commercial energy consumption (Table 19) | 6.9 | 54.3 |
| Delivered industrial energy consumption (Table 20) | 11.3 | 82.4 |
| Delivered transportation energy consumption (Table 21) | 10.3 | 77.6 |
| Total energy related carbon dioxide emissions (Table 22) | 12.8 | 78.1 |
| Energy intensity (energy consumption / real $ GDP) (Table 23) | 12.3 | 97.7 |
| AEO – Annual Energy Outlook. Source: These statistics summarize the calculations in Tables 2 through 23. The data in Tables 2 through 23 are based on the 1994 through 2020 AEO Reference case projections. Historical Data are from the U.S. Energy Information Administration open data API (http://www.eia.gov/opendata/) (Washington, DC, September 2020), with the series listed under each appendix table, except for GDP data which are from the Bureau of Economic Analysis, U.S. Dept. of Commerce, August 2020, http://www.bea.gov/national/xls/gdplev.xlsx 1 The basis for GDP comparison is the projection differences in the cumulative average growth rate of real GDP from the first year shown for each AEO. The summary information for projection differences given for GDP growth rates is absolute percentage point differences; for all other AEO concepts, the comparison basis is absolute percent differences. 2 As of 2013, the wellhead price of natural gas was no longer reported by EIA. With the 2015 edition of the Retrospective, the natural gas price to the electric power sector replaced the wellhead price. 3 As natural gas net imports approached zero and then turned negative (indicating the change to U.S. net exporter status), the average absolute percent difference increased significantly. 4 Beginning in AEO2003, EIA electric generating projections incorporated combined heat and power (CHP) electricity generation in electricity generating plants. Prior to AEO2003, coal price projections reflected data collected, estimated, and reported to electric utilities and excluded CHP power generation. |
||
Notes
1. NEMS documentation is available on the EIA website.
2. EIA has used the National Energy Modeling System to prepare the AEO since 1994. This publication considers only the projections made after 1994. In addition, the Annual Energy Outlook 2009 results are from “An Updated Annual Energy Outlook 2009 Reference Case Reflecting Provisions of the American Recovery and Reinvestment Act and Recent Changes in the Economic Outlook,” which is available on the EIA website. Further, projections in AEO1994 and AEO1995 ended in 2010, so entries for these two publications are blank for the later years in Tables 2 through 22.
3. The exceptions include historical data related to gross domestic product from the U.S. Department of Commerce, Bureau of Economic Analysis (BEA).
4. The methodology used is derived from Kaack, et al. “Empirical prediction intervals improve energy forecasting,” Proceedings of the National Academies of Sciences, 114(33) published August 15, 2017.
5. H is defined as the number of years since the first projection year, which varies by AEO (for example, H=0 corresponds to the year 2017 for AEO2018 and to the year 2016 for AEO2017).
6. Although dynamic feedback in the model can modify these initial forecasts, the resulting changes tend to be small.