Evaluation of 2016 and Prior Reference Case Projections
The U.S. Energy Information Administration (EIA) produces projections of the United States’ energy production, consumption, and prices each year in the Annual Energy Outlook (AEO). The AEO is produced on an alternating cycle, with a full AEO (including complete documentation and a large number of alternative cases) followed by a shorter AEO (more limited documentation and only a few alternative cases). Each year a full AEO is released, EIA also produces the AEO Retrospective Review (AEO Retrospective). The AEO Retrospective compares recent history with the Reference case projections from previous editions of the AEO. The AEO Retrospective shows the relationship between past AEO projections and actual energy indicators, and informs discussions of the underlying models.
The projections presented in the AEO are not statements of what will happen but of what may happen given the assumptions in the underlying National Energy Modeling System (NEMS). These assumptions include projections of oil prices and gross domestic product (GDP), and are publically available.1 The AEO Reference case projection generally assumes that other trends are consistent with historical and current market behavior, technological and demographic changes, and current laws and regulations.
Although the AEO Retrospective focuses on the Reference case projections, readers are encouraged to review the full range of cases in the AEO, which illustrate the uncertainty inherent in long-term projections. Beyond the Reference case, the discussion and analysis in each year’s AEO typically include Low and High Economic Growth cases, and Low and High Oil Price cases. Recent AEOs have also included alternative cases examining the impact of proposed policy changes, variation in the rate of technology improvement, and different assumptions about the size of the energy resource.
The AEO Retrospective summarizes the relationship between the Reference case projections and realized energy usage by comparing absolute differences and the average absolute percent differences for many of the major measures published in AEOs from 1994 to 2016.2 The average absolute percent difference is the simple mean of the absolute values of the percentage difference between the Reference case projections and the actual values. Most historical data are from the Monthly Energy Review (MER), which can be accessed from the EIA open data API.3
Table 1 provides a summary of 25 comparisons based on 21 primary measures projected in the AEO. Tables 2 through 22 show the detailed results which include the Reference case projections, historical values, and percent differences between projected and actual values for all covered years. The detailed tables also provide the average absolute difference across all AEOs for each covered year. Prior-year statistics can change from one year’s evaluation to the next because of updates to historical data in the MER and changes in the measurement of GDP. The statistics in Table 1 do not account for the continual improvements made to NEMS in an attempt to more accurately capture developing energy market trends based on more recent experience.
The first column in Table 1 presents the average absolute percent difference for each projected measure over the period of 1994–2016. The second column of Table 1 provides a summary of the percentage of cases in which a particular measure was overestimated relative to its actual value out of the total number of annual projections for which an actual value can be compared. A zero represents a measure that was always underestimated, and 100% would represent a measure that was always overestimated. In an unbiased projection, with a sufficiently large number of samples, overestimates would constitute approximately 50% of the estimates.
As shown in Table 1, measures of energy consumption are projected with greater accuracy than measures of net imports and energy prices. Energy consumption tends to change at a slower pace than other indicators; many energy-using devices, such as appliances, automobiles, and industrial equipment, are expensive to purchase and have long service lives. The substantial lead time for these large purchases, along with the effects of energy contracts and other mechanisms, tend to slow changes in consumer energy demand.
Table 1 illustrates the relative accuracy of projected energy demand and energy price. The average absolute percent differences for energy consumption, production, and carbon dioxide emissions range roughly between 5% and 15%. By contrast, the absolute percent differences for energy prices range between 10% and 40%. Net imports tend to exhibit even larger average differences, ranging between 40% and 90%.4
For the measures with wider deviations, many factors contribute to differences between the AEO Reference case projections and realized outcomes, but two primary contributors are the initial projections of future oil prices and overall economic activity that are used in NEMS.5 These projections can greatly influence the other projections made by the model, which is why each recent AEO includes alternative cases exploring differences in economic growth (Low and High Economic Growth cases) and in oil prices (Low and High Oil Price cases).
Changes in industry-specific market conditions, significant technological breakthroughs, and new laws or regulations subsequent to the publication of an AEO can also lead to differences between projections and realized outcomes.
| Variable | Average Absolute Percent Differences | Percent of Projections Over- Estimated |
|---|---|---|
| Gross Domestic Product | ||
| Real Gross Domestic Product (Average Cumulative Growth)* (Table 2) | 0.9 | 52.6 |
| Petroleum | ||
| Imported Refiner Acquisition Cost of Crude Oil (Constant $) (Table 3a) | 39.7 | 21.4 |
| Imported Refiner Acquisition Cost of Crude Oil (Nominal $) (Table 3b) | 38.1 | 22.9 |
| Total Petroleum Consumption (Table 4) | 9.0 | 72.6 |
| Crude Oil Production (Table 5) | 11.9 | 43.2 |
| Petroleum Net Imports (Table 6) | 43.0 | 77.1 |
| Natural Gas | ||
| Natural Gas Price, Electric Power Sector (Constant $)** (Table 7a) | 30.9 | 39.5 |
| Natural Gas Price, Electric Power Sector (Nominal $)** (Table 7b) | 30.5 | 42.1 |
| Total Natural Gas Consumption (Table 8) | 8.2 | 64.7 |
| Natural Gas Production (Table 9) | 8.1 | 52.6 |
| Natural Gas Net Imports (Table 10) | 87.4 | 77.8 |
| Coal | ||
| Coal Prices to Electric Generating Plants (Constant $)*** (Table 11a) | 20.5 | 34.2 |
| Coal Prices to Electric Generating Plants (Nominal $)*** (Table 11b) | 20.7 | 37.6 |
| Total Coal Consumption (Table 12) | 14.5 | 71.4 |
| Coal Production (Table 13) | 10.7 | 73.3 |
| Electricity | ||
| Average Electricity Prices (Constant $) (Table 14a) | 10.7 | 28.9 |
| Average Electricity Prices (Nominal $) (Table 14b) | 12.0 | 35.3 |
| Total Electricity Sales (Table 15) | 5.5 | 59.8 |
| Total Energy, Carbon and Intensity | ||
| Total Energy Consumption (Table 16) | 7.6 | 81.2 |
| Delivered Residential Energy Consumption (Table 17) | 5.7 | 70.3 |
| Delivered Commercial Energy Consumption (Table 18) | 6.7 | 54.1 |
| Delivered Industrial Energy Consumption (Table 19) | 11.6 | 85.3 |
| Delivered Transportation Energy Consumption (Table 20) | 9.1 | 80.8 |
| Total Energy Related Carbon Dioxide Emissions (Table 21) | 10.4 | 72.9 |
| Energy Intensity (Energy Consumption / Real $ GDP) (Table 22) | 9.6 | 86.8 |
| AEO – Annual Energy Outlook. Source: These statistics summarize the calculations in Tables 2 through 22. The data in Tables 2 through 22 are based on the 1994 through 2016 AEO Reference case projections. Historical Data are from the U.S. Energy Information Administration open data API (http://www.eia.gov/opendata/) (Washington, DC, June 2016), with the series listed under each appendix table, except for GDP data which are from the Bureau of Economic Analysis, U.S. Dept. of Commerce, June 2016, http://www.bea.gov/national/xls/gdplev.xls. * The basis for GDP comparison is the projection differences in the cumulative average growth rate of real GDP from the first year shown for each AEO. The summary information for projection differences given for GDP growth rates is absolute percentage point differences; for all other AEO concepts, the comparison basis is absolute percent differences. ** As of 2013, the wellhead price of natural gas was no longer reported by EIA. With this edition of the Retrospective, the natural gas price to the electric power sector, which was projected in each of the AEOs being evaluated, replaces the wellhead price. *** Beginning in AEO2003, EIA electric generating projections incorporated combined heat and power (CHP) electricity generation in electricity generating plants. Prior to AEO2003, coal price projections reflected data collected, estimated, and reported to electric utilities and excluded CHP power generation. |
||
The following points highlight the main findings from the more detailed tables:
- Two broad trends are shown in the AEO projections of macroeconomic growth. AEOs published between 2001 and 2009 generally over-projected macroeconomic growth, largely as a result of the 2007–2009 financial crisis. Earlier AEOs systematically under-projected economic growth, largely because of the robust 3.9% average growth in real GDP during the period of 1993–2000.
- While the refiner acquisition cost of imported crude oil in 2013 was just under $100 per barrel, down nearly $5 from its peak in 2011, many AEOs published prior to 2006 projected prices less than $35 per barrel by 2014 (Tables 3a, 3b). The actual acquisition cost declined to $47 per barrel by 2016, and AEO Reference cases published after 2005 over-projected this price.
- Petroleum net imports peaked in 2005, falling successively in each subsequent year, but most AEOs published before 2006 projected net import growth continuing beyond 2005. In contrast, AEO Reference cases published after 2006 correctly projected net imports to decline, in part because of the subsequent economic downturn and increasing vehicle fuel efficiency standards (Table 6).
- Coal consumption is dominated by use for electric generation, which grew modestly until 2009, and coal prices to electric generating plants, which increased since 2000 until they flattened in 2009 (Tables 11a, 11b, 12). The natural gas price to the electric power sector also dropped nearly 50% primarily because of increases in natural gas production in 2009 (Tables 7a, 7b). As a result, 2009 coal consumption dropped by more than 10% and has not subsequently recovered. These unanticipated developments led to coal consumption being over projected in AEO Reference cases after 2008 (Table 12).
- Natural gas production increased each year since 2005, and it reached a record-high level in 2015 (Table 9). Natural gas net imports peaked in 2007, falling successively in each year since, with 2015 at the lowest level over the AEO years shown (Table 10). AEOs published after 2006 have tended to underestimate natural gas production and consequently overestimate net imports.
- Electricity prices were almost always underestimated in the AEO Reference cases published after 1997 (Tables 14a, 14b). In deregulated markets, the price of natural gas tends to determine marginal electricity prices. With part of the United States remaining regulated, projecting a national electricity price is complex. Regardless, the underestimation of natural gas prices before 2009 and of coal prices in later periods led to an under projection of electricity prices.
- U.S. energy consumption peaked in 2007, but nearly all AEO Reference cases projected rising consumption after this point because these cases did not anticipate the strong effects of the 2008-2009 recession and its relation to reductions in energy intensity. Although the differences between the related projections and actual outcomes were small, some pattern of overestimation is evident (Tables 16 to 20).
- Projected energy intensity is the concept most often over projected (Table 22) in the AEO. Energy intensity is defined as the ratio of total energy consumption to real GDP, so differences between the related projections and actual outcomes depend on differences in both energy consumption and real GDP. For AEOs prior to 2001, energy intensity tended to be over-projected mainly because of the under-projected GDP (see Table 1). For AEOs published after 1998, energy consumption was over-projected, leading to an over-projection of energy intensity.
Notes
1NEMS documentation is available on the EIA website at: https://www.eia.gov/outlooks/aeo/nems/documentation/
2The National Energy Modeling System has been used to prepare the AEO since 1994. This publication considers only the projections made after 1994. In addition, the Annual Energy Outlook 2009 results are from “An Updated Annual Energy Outlook 2009 Reference Case Reflecting Provisions of the American Recovery and Reinvestment Act and Recent Changes in the Economic Outlook,” which is available on the EIA website at: https://www.eia.gov/analysis/requests/archive/1999/stimulus/. Further, projections in the 1994 and 1995 AEOs ended in 2010, so entries for these two publications are blank for the later years in Tables 2 through 22.
3 The exceptions include historical data related to Gross Domestic Product from the U.S. Department of Commerce, Bureau of Economic Analysis.
4This finding is primarily influenced by the results of AEOs prior to 2008, which tended to over-project net imports by under-projecting decreases in natural gas imports.
5Athough dynamic feedback in the model can modify these initial forecasts, the resulting changes tend to be small.