|
Annual
Energy Outlook
Forecast Evaluation
by Esmeralda
Sanchez
Errata -(7/14/04)
The Office of Integrated Analysis and Forecasting has produced an annual
evaluation of the accuracy of the Annual Energy Outlook (AEO) since 1996.
Each year, the forecast evaluation expands on the prior year by adding
the projections from the most recent AEO and the most recent historical
year of data. The Forecast Evaluation examines the accuracy of AEO forecasts
dating back to AEO82 by calculating the average absolute forecast errors
for each of the major variables for AEO82 through AEO2003. The average
absolute forecast error, which for the purpose of this report will also
be referred to simply as “average error” or “forecast
error”, is computed as the simple mean, or average, of all the absolute
values of the percent errors, expressed as the percentage difference between
the Reference Case projection and actual historic value, shown for every
AEO and for each year in the forecast horizon (for a given variable).
The historical data are typically taken from the Annual Energy Review
(AER). The last column of Table 1 provides a summary of the most recent
average absolute forecast errors. The calculation of the forecast error
is shown in more detail in Tables
2 through 18. Because data for coal prices to electric generating
plants were not available from the AER, data from the Monthly Energy Review
(MER), July 2003 were used.
The 2003 Forecast Evaluation adds an evaluation of Energy Intensity (Table
18) and eliminates the evaluation of Net Coal Exports. During the past
few years, the Energy Information Administration (EIA) has seen increased
public interest in energy intensity, particularly as public policy issues
such as carbon dioxide emissions, technological development, impacts of
structural changes on the economy, and national energy security, are more
openly discussed and evaluated. Given this increased interest, EIA believed
energy intensity consumption to be important variables to examine. For
purposes of the Forecast Evaluation, energy intensity is defined as the
ratio between two output variables of the model: total energy consumption,
measured in British thermal units (Btu), and gross domestic output (GDP)
in nominal dollars. The decision to exclude Net Coal Exports (Table
12 in prior Forecast Evaluations) from current and future Forecast Evaluations
reflects the relatively small role that coal exports play in overall U.S.
energy trade.
The underlying reasons for deviations between the projections and realized
history tend to be the same from one evaluation to the next. The most
significant are as follows:
- Over the last two decades, there have been many significant changes
in laws, policies, and regulations that may or may not have been predictable
and were not assumed in the projections prior to their implementation
based on EIA’s statutory requirement to be policy neutral. Many
of these actions have had significant impacts on energy supply, demand,
and prices.
- While energy price forecasts in the AEOs published in years 1994 to
2003 have improved, petroleum, natural gas, coal, and electricity price
projections continue to be more difficult to accurately forecast than
consumption, production, and net imports. Energy price forecasts are
done in real prices, but for purposes of the Forecast Evaluation, they
are re-inflated to nominal dollars using the forecast deflator of a
given AEO forecast and compared to the nominal prices listed in the
AER. As a general rule, the rate of increase in nominal energy prices
has been overestimated rather than underestimated. More rapid technological
improvements, the erosion of the market power of the Organization of
Petroleum Exporting Countries starting in the mid-1980s, excess productive
capacity, and market competitiveness are all factors that led to lower
energy prices than projected. In the 1980s and 1990s, productivity and
technology improvements, as well as the effects of gradual deregulation
and changes in industry structure, have offset the factors known to
result in higher energy prices, such as resource depletion and growing
energy demand. However, the tendency to overestimate energy prices may
be stabilizing or even reversing. In more recent years, better adherence
by the oil-producing countries to crude oil production quotas, evidence
of increasing rates of depletion in producing fields (natural gas and
oil), short-term supply constraints (natural gas), or the unexpected
consequences of deregulation policies (electricity), have led to generally
higher prices than expected only a few years ago.
- Technological improvements in both the production and use of energy
have had a significant impact on the price, supply, and consumption
of energy. For the most part, earlier AEOs assumed much slower technology
development than actually occurred, accounting for some of the deviation
between the forecasts and history. This influence can be seen in the
forecast evaluation tables.
- The level of future electricity sales has been consistently underestimated
in all years in each of the AEOs published between 1994 and 2001. The
underestimation, which averaged less than 3 percent, implies that the
fuels consumed to generate electricity (e.g., coal, oil, natural gas)
were also underestimated. Forecasting errors related to electricity
sales have increased from 1.7 percent in the 1998 Evaluation to 2.4
percent in this year’s Evaluation. These errors are, primarily,
driven by electricity sales growth projections that proved conservative
in AEO1994, AEO1995, and AEO1996 for the years 1996 through 2002. Electricity
sales increased 12 percent during this period. There have been many
changes in the structure of electricity markets and growing sources
of uncertainty in the industry over the last decade may help to explain
this underestimation. Such factors include deregulation of electric
generation, the proliferation of computer equipment and consumer electronics,
growth in Internet use, and the expansion of the information technology
sector during the economic boom of the 1990s.
- The underestimation of electricity growth over the 1996 to 2002 forecast
period had an important impact on the coal consumption forecasting error
over this period. About 90 percent of the demand for coal results from
electricity generation and an underestimation of electricity sales results
in an underestimation of coal consumption. As a consequence, coal consumption
has consistently been underestimated in the AEO. The average absolute
percent error for the forecast of coal consumption has ranged between
3 and 4 percent.
- Overestimation of world oil prices, particularly in earlier AEO publications,
resulted in upward pressure on production/import forecasts, while underestimation
of exploration and production technological improvement put downward
pressure on production forecasts. In addition, high world oil price
forecasts tended to lower oil consumption forecasts relative to what
actually occurred. This problem was exacerbated by the unexpected shift
in consumer preference to larger vehicles.
- Natural gas generally has been the fuel with the least accurate forecasts
in consumption, production, and prices. As regulatory reforms that increased
the role of competitive markets were implemented beginning in the mid-1980s,
the behavior of natural gas in competitive markets was especially difficult
to predict. In the earlier forecasts, EIA’s technology improvement
expectations proved conservative, as technological advances made natural
gas less costly to produce, while in the more recent forecasts, EIA
overestimated technology improvements. After natural gas curtailments
were eased in the mid-1980s, environmental pressures made natural gas
an increasingly attractive fuel source, particularly for electricity
generation. Historically, natural gas price instability was strongly
influenced by natural gas resource estimates, which steadily rose, and
by the world oil price, which was subject to its own error. Despite
growing uncertainty regarding the ability of future natural gas production
to meet demand, natural gas wellhead price projections, as shown in
Table 1, have steadily improved from an average absolute percent error
of 72.1 in the 1998 Evaluation to 57.3 percent in the 2003 Evaluation.
- External factors such as severe weather, economic cycles, and strikes
have also had an impact on energy markets; however, these events cannot
be anticipated in the mid- to long-term period and are not captured
in the models underlying the AEO projections.
- The carbon dioxide emissions error increased from 1.8 percent to
2.6 percent in 2003. This error is linked to overestimation of GDP growth
during for years 2000 to 2002 in AEO2001 to AEO2003.
Table 1 shows the forecast
errors from previous and current forecast evaluations. As indicated in
Table 1, the forecasts of consumption, production, and carbon dioxide
emissions have generally been the most accurate while the forecasts of
prices and net imports have been less accurate. The percent errors change
from one year’s evaluation to the next as an additional year of
data and projection is added. The percent errors may also change due to
data revisions in the AER and the MER.
Both the passage of time and the lengthening of the maximum AEO projection
horizon1 over the 22 years that the
AEO has been published mean that the average interval between issuance
of a projected quantity and the realization of the actual value to which
it is compared for forecast evaluation purposes is increasing with successive
editions of this document. For example, for 1985 data, the average interval
between forecast and actual values is 2 years. In contrast, for the 2002
data, which is new to this year’s evaluation, the average interval
between forecast and realization is 6 years. Because sources of error
and uncertainty can be expected to increase as the forecast horizon lengthens,
one might expect forecast performance measured in terms of average absolute
error to degrade as the share of long-horizon projections in the forecast
evaluation grows. Improvements in forecasting methods, however, work in
the other direction, and the results reported in Table 1 reflect both
of these effects.
An examination of the data in the last column in Table 1 suggests an
improvement in some AEO forecast categories, particularly in forecasts
of natural gas production, net petroleum imports, world oil prices, and
natural gas wellhead prices. The improvement in the forecasts of world
oil prices and natural gas wellhead prices is particularly striking. The
average absolute percent error in natural gas wellhead prices decreased
from 72.1 percent in the 1998 Forecast Evaluation to 57.3 percent in the
2003 Forecast Evaluation. The error in world oil prices decreased from
51.3 percent in the 1998 Evaluation to 48.9 percent in the current Evaluation.
On the other hand, a number of energy consumption categories show higher
errors, including total energy, natural gas, and coal. For the most part,
the percent error remains similar or improves over time.
1The forecast horizon of AEO82 was 8 years.
AEO86 increased the maximum horizon for annual forecasts to 15 years,
while AEO98 extended it to 22 years.
|