Energy Policy 61 (2013) 360–370

Contents lists available at ScienceDirect

Energy Policy
journal homepage: www.elsevier.com/locate/enpol

Engineering estimates versus impact evaluation of energy efficiency

projects: Regression discontinuity evidence from a case study
Corey Lang n, Matthew Siler
Department of Environmental and Natural Resource Economics, University of Rhode Island, 214 Coastal Institute,
1 Greenhouse Rd., Kingston, RI 02879, United States

H I G H L I G H T S

Regression discontinuity used to estimate energy savings from efficiency projects.

Ex post econometric estimates validate ex ante engineering estimates of energy savings.

Select efficiency projects shown to reduce peak load.

art ic l e i nf o a b s t r a c t

Article history: Energy efficiency upgrades have been gaining widespread attention across global channels as a cost-
Received 12 March 2013 effective approach to addressing energy challenges. The cost-effectiveness of these projects is generally
Accepted 23 June 2013 predicted using engineering estimates pre-implementation, often with little ex post analysis of project
Available online 16 July 2013
success. In this paper, for a suite of energy efficiency projects, we directly compare ex ante engineering
Keywords: estimates of energy savings to ex post econometric estimates that use 15-min interval, building-level
Energy efficiency energy consumption data. In contrast to most prior literature, our econometric results confirm the
Engineering-economic evaluation engineering estimates, even suggesting the engineering estimates were too modest. Further, we find
Peak load heterogeneous efficiency impacts by time of day, suggesting select efficiency projects can be useful in
reducing peak load.
& 2013 Elsevier Ltd. All rights reserved.

1. Introduction (2012) argue that the gap is likely very small, in the order of 1–2% of
energy use.
Improving energy efficiency is an increasingly important com- One of the sticking points at the heart of this debate centers on
ponent of energy policy in the United States and around the world. discrepancies between engineering estimates and econometric
As a result, substantial resources are being funneled into energy impact evaluations of energy savings. Dubin et al. (1986) and
efficiency projects and programs. For instance, two-thirds of the Nadel and Keating (1991) both found engineering estimates to be
$825 million in revenue generated by Regional Greenhouse Gas greatly overstated compared to ex post measurements, and
Initiative in the Northeast United States through permit auctions hypothesized the discrepancy was due to price effects and inac-
since its inception through 2011 was directed into energy effi- curate engineering estimation techniques. These early studies have
ciency programs (RGGI Inc., 2012). proven highly influential in shaping economists' and others'
While many believe that improving energy efficiency is a cost perceptions of this debate, and there is a dearth of similar studies
effective way of reducing energy consumption, others (often despite many years of programs since.
economists) have found little empirical evidence of unexploited, This paper seeks to augment the literature comparing engi-
profitable investments in energy efficiency—the so-called energy neering and econometric estimates by examining a case study for
efficiency gap. Granade et al. (2009), in a report for McKinsey & which we have ex ante engineering estimates of savings and we
Company, describe energy efficiency as a vast, low-cost resource can estimate ex post savings using building-level energy use
worth over $1.2 trillion if upfront investment in efficiency measures (“smart meter”) data. Specifically, we examine three lighting
through 2020 were executed at scale. However, Allcott and Greenstone equipment upgrades undertaken at the Naval War College in
Newport, Rhode Island. In October 2009, Secretary of the Navy
Ray Mabus announced five ambitious energy targets, and as result
n
Corresponding author. Tel.: +1 401 874 4569; fax: +1 401 874 1256. the Navy is now paying close attention to the energy factors of its
E-mail address: [email protected] (C. Lang). operations (Mabus, 2009). Among the targets was a goal to ensure

0301-4215/$ - see front matter & 2013 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.enpol.2013.06.122
 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370 361

that at least 40% of the Navy's total energy consumption comes counterparts (Dubin et al., 1986; Nadel and Keating, 1991; Joskow
from alternative sources by the year 2020. The high price points of and Marron, 1992; Metcalf and Hassett, 1999). While we only
current alternative energy sources led naval institutions to first present evidence from a case study, it is an important benchmark
minimize existing energy footprints.1 To this end, over the past that these two types of estimates can indeed match. One possibi-
three years the Naval War College has been implementing various lity to explain consistency in estimates here but not in prior papers
small scale projects – primarily lighting equipment upgrades – to is that in this case there is not a significant possibility for a
reduce their electrical load. In addition to upgrading to more countervailing behavioral response. Further, there may have been
efficient bulbs, the suite of projects implemented at the War methodological improvements over the past 25 years in the way
College employ technologies such as occupancy sensors, ambient engineers estimate savings (for our case study, these methods are
light sensors and timers in order to reduce usage. described in Section 2). However, it is also likely easier to estimate
To empirically measure electricity consumption changes, we savings from lighting compared to heating and cooling because
collected smart meter recordings of kilowatt hours (kW h) in leakage is not an issue.
15 min intervals over the span of 15 months for each project A second contribution of this paper is to highlight methods and
building on the Naval War College campus. The high frequency data for calculating energy savings from energy efficiency invest-
nature of the data enables two critical aspects of the present ments. When there is a discrete time of implementation, regres-
research. First, we are able to implement a regression discontinu- sion discontinuity is an ideal framework because identification of
ity (RD) research design, with time as the forcing variable and the the treatment effect allows for unobserved variables—of which
time of installation marking the discontinuity, to estimate reduc- there are many when it comes to energy use. This method lends
tions in electricity demand caused by lighting upgrades. Given that itself particularly well with high frequency readings from smart
energy use is a function of many unobserved factors, RD is ideal for meters, which are becoming increasingly available and affordable.
measuring the impacts of an installation. Further, RD, in conjunction with difference-in-differences or
The second critical aspect that the 15 min interval data allow is another approach that can examine long term adjustments, may
an examination of how project impacts vary over the course of a prove valuable for disentangling immediate impacts versus beha-
day. While reducing total energy consumption is perhaps the most vioral changes that occur at a lag.
frequent goal of energy policy, reducing peak demand is another Lastly, our paper may contribute to the growing literature and
valuable objective. On a typical summer New England day, Joskow understanding of the energy efficiency gap. Because the engineer-
(2012) reports peak load production costs being over six times ing and econometric estimates align, it appears that prior to the
greater than at base load.2 It is unknown to what extent the two implementation of these three energy efficiency projects, there
goals of reducing total consumption and reducing peak load can be were unexploited, profitable investments that the Naval War
simultaneously achieved through energy efficiency projects, or if College had at its disposal. One of the avenues by which the
there are substantial tradeoffs, and our case study addresses this. energy efficiency gap exists is through imperfect information.
Our results suggest that measurable reductions in energy Anecdotally, it seems that the ambitious energy agenda set forth
consumption can be achieved through simple lighting upgrades by Secretary Mabus initiated facilities managers to understand
as demonstrated by three energy efficiency projects implemented energy use better and seek out projects for reductions. However,
and tracked at the Naval War College. The econometric results there may be hidden costs such as administrative or managerial
confirm the engineering estimates, even suggesting the engineer- time or inconvenient changes in energy services that caused the
ing estimates were too modest for two of the three projects. Naval War College not to adopt these efficiency improvements
We attribute the additional energy savings to behavioral spillovers earlier, and our data and analysis cannot speak to these hidden
in other parts of the building, such as employees and students costs.3
being more mindful of turning off lights and computers, though The paper proceeds as follows. In Section 2, we describe in
we are unable to test this hypothesis. detail each of the three projects undertaken at the Naval War
Our analysis of heterogeneous impacts by hour of day finds that College, discuss key information about our data, and outline the
energy reductions can vary substantially throughout the day. For empirical models used. Section 3 presents results and compares
one of the projects that we analyze, monetizing peak load the ex ante engineering estimates with our ex post data driven
reductions leads to a 25% increase in estimated benefits of the estimates. Section 4 concludes with generalized discussion of the
project compared to the estimated benefits using a flat electricity results and their implications.
rate. While current research tends to emphasize the importance of
dynamic pricing (e.g., Wolak, 2011) and load management (e.g.,
Newsham et al., 2011) to reduce peak demand, our results suggest
that certain efficiency projects can also address peak load. We 2. Setting, data, and methodology
discuss in detail the characteristics of these lighting projects that
are determinants of peak demand reductions. 2.1. Description of projects
The main contribution of this paper is to offer new evidence on
the relationship between engineering and econometric estimates In response to Secretary of the Navy Ray Mabus' aggressive
of energy savings. Our results stand in contrast to the findings of energy reduction goals, the Naval War College of Newport, Rhode
many papers that find ex ante engineering estimates of benefits to Island implemented a series of three energy efficiency projects. Each
be overstated when compared to their empirically derived ex post of the projects sought to improve efficiency via enhanced lighting
technologies, lamp upgrades or through the implementation of

1 3
Having achieved many reductions in energy use, the Naval War College is Gillingham et al. (2009) identify and discuss a host of potential market and
now planning to build 9 MW of wind capacity on or near campus to continue their behavioral failures that are attributable to the existence of the energy efficiency
renewable energy goals (Hence, 2013). gap. They offer a nice discussion of how investments can appear profitable to
2
Seasonal and daily peak load require the existence of power plants that only outsiders, but unmeasured physical costs, risks and opportunity costs may lead to
produce during those times of need. Beyond the inefficiencies of having capital that non-adoption. As an example of this, Anderson and Newell (2004) analyze
is rarely in use, peak load is when the costs of supplying electricity as well as technology adoption decisions of manufacturing firms following energy audits
marginal CO2 emissions per kilowatt hour are highest. Thus, there is an economic and find that the firms adopt only half of the recommendations due to factors
and a social imperative to reduce peak load consumption. unaccounted for by the auditors.
362 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370

Table 1
Project summaries.
Source: Authors' data.

Project description Average 15-min kW h Engineering estimates

reading pre-installation
Installation Annual Annual Payback
costs kW h dollars period
savings saved (years)

(A) Library: 27 library stacks were populated with infrared occupancy sensors. Pre- 51.00 $5,492 42,357 $4,725 1.32
implementation, lamps remained lit. Post-implementation, lamps within a bookstack
are lit only when human presence is sensed within the stack. The project was
implemented July 26–30th, 2010

(B) Office: A small office was populated with occupancy sensors. Excessive lighting was 5.21 $6,605 9,223 $1,029 7.31
removed, and remaining lighting was reallocated to provide an even balance. The
project was implemented Aug. 2–6th, 2010

(C) Hallway: A long hallway with many windows was populated with occupancy and 20.33 $6,768 42,362 $4,726 1.63
daylight sensors. Pre-implementation, lamps remained lit. Post-implementation, lamps
will energize for 15 min when the hallway is both occupied, and internal lighting
conditions including sunlight from windows, are dimmer than the Navy lighting
standard. The project was implemented Aug. 21–22nd, 2011

occupancy and daylight sensors. Refer to Table 1 for a summary of electricity rate of 11.1 ¢/kW h.4 This project was implemented July
the three projects undertaken. 26th–30th, 2010.
To complete cost benefit analyses of these three projects, the
facilities engineer led an engineering study of energy savings.
2.1.2. Project B
The engineering estimation technique evaluated existing lighting
The second project involved an overhaul of the lighting systems
loads directly by counting fixtures in the project areas, and
within a 17,700 square foot office building. This project entailed
recording the wattage of those fixtures. Average daily usage was
the removal of excessive lighting (140 lamps; 4296 kW under load)
estimated by planting logging devices that continuously moni-
as well as a reallocation of remaining lighting to maximize balance
tored lighting activity throughout the week. Occupancy loggers
and occupant comfort.5 Also included in the lighting upgrades was
were also used to map human activity in the project spaces
the installation of a series of occupancy sensors. The sensors
throughout the day. All three lighting conservation projects called
employed infrared and ultrasonic technologies to switch lamps
for occupancy sensors to switch lighting, so ex ante savings
on as spaces became occupied, and hold them in that state for
estimates were calculated by finding the average time difference
15 min after being triggered. Sensors were installed in strategic
between current lighting activity, and actual occupancy trends.
locations to ensure intended function while minimizing the
One project also incorporated photo sensing technology in the
occurrence of false triggers. Installation costs for this project were
switching solution alongside motion sensors. For this project,
$6605, which were relatively high given the scale of this project
estimated savings also considered the temporal windows where
due to more electrician labor hours. Ex ante engineering estimates
spaces were occupied, but external lighting through windows was
indicated an annual savings of 9223 kW h and $1029. This project
sufficiently high enough to safely hold internal lighting off. In
was implemented August 2–6, 2010.
another project where some fixtures were removed, these changes
to maximum load were taken into account. Finally, the appropriate
electrical rate was applied to calculate cost savings. 2.1.3. Project C
This project improved lighting infrastructure in a long walkway
connecting three of the college's buildings, and was designed to
take advantage of the fact that the walkway was well lit on sunny
2.1.1. Project A days via numerous floor-to-ceiling windows. Despite the abundant
The first project implemented was a lighting upgrade in a large daylight during most work hours, the hallway was lit continuously
section of the college's library. The lighting within 27 library stacks by 195 four foot fluorescent bulbs. Further, there was no ability to
was automated using infrared occupancy sensors. This was an shut off the lights other than through a secured breaker box.
ideal location for implementing motion sensors for several rea- The energy efficiency upgrade selected to address these issues
sons. First, routine observation suggested that occupants traffic featured two technologies. First, standard occupancy sensors send
this particular section of the library infrequently. Furthermore, a trigger for lamps to switch on when human presence is detected.
banks of lighting were not being turned off before workers These lamps remain lit for 15 min to ensure safe passage through-
returned home in the evenings, before weekends or even long out the walkway during evening hours. Second, a daylight sensor
holidays. Infrared sensors with a narrow viewing angle and long
range were installed on opposite sides of each library stack. Chains 4
The rate of 11.1 ¢/kW h does not reflect the true rate the Naval War College
of fluorescent tube lighting contained within each stack would
has negotiated, which is currently at 9.8 ¢/kW h. We use 11.1 ¢ instead because this
then switch on only after that particular stack was accessed by an is the average cost of the production cost curve presented in Joskow (2012), which
individual either viewing or returning a book. The lights would we importantly use to value energy reductions at different times of day. In order for
remain on while the stack was occupied, and switch off 1 min after our comparisons to be on equal footing, we use 11.1 ¢.
5
the stack was vacated. One time installation costs of $5492 for this Instead of just being about energy efficiency, this project may have changed
the energy services and theoretically could have reduced worker utility. However,
project primarily constitute electrician labor, but also include the given our knowledge of the lighting before and after and anecdotal conversations
cost of the new sensors. Ex ante engineering estimates predicted with employees, we are confident that worker utility was unlikely to decline and
an annual savings of 42,357 kW h and $4725, using a flat may have increased.
 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370 363

was installed, which blocked the switching trigger if ambient light coefficient of interest as it estimates the average kW h savings per
from the windows met the safe-lighting level standard without 15 min interval due to the project. Next in the equation is a polynomial
the need for artificial lighting. Installation costs for this project function of time, f ðtÞ, where
were $6768, which again was due mostly to electrician labor. P P
Ex ante engineering estimates predicated an annual savings of f ðtÞ ¼ ∑ γ 1k  t k þ ∑ γ 2k  t k  post_treatment t ð2Þ
42,362 kW h and $4726.6 The sensors for this project were k¼1 k¼1

installed between August 21st–22nd, 2011. However, calibration We include this term to flexibly capture time trends in kW h
of the multiple types of sensors persisted through September, with before and after implementation. The time variable was created
small adjustments made as late as December. such that time is continuous before and after implementation of
the project, and the days in which the project was undergoing
2.2. Data implementation have been removed. Further, we define time such
that the project was completed at time zero, which guarantees
The critical data needed to measure the impacts of the energy that the full treatment effect will be captured by β since the second
efficiency projects described in Section 2.1 are accurate measures set of terms in Eq. (2) will equal zero. Since each project was
of energy use before and after the implementation of the projects. implemented at a different time, we essentially have different time
While energy use data for just the section of buildings targeted by variables for each project. We estimate models for both P ¼3 and
the projects were not available, we were able to attain energy P¼ 4, i.e., cubic and quartic time trends before and after the
consumption data at the building level. In-situ recording devices installation, to allow flexibility and test robustness of results.
were built into the electrical meters feeding each project building Lastly, Eq. (1) includes a vector of controls, X; to combat several
on the Naval War College's campus. These smart meters recorded potentially confounding factors. First, X includes a cubic polyno-
energy consumed, in kW h, over 15 min intervals, for every 15 min mial for daily average temperature. While the energy projects
interval in a day, 7 days a week. This energy consumption data is examined have nothing to do with heating or cooling, our
available for each project building from May 27, 2010 (the date of dependent variable captures energy use for an entire building,
smart-meter installation) until March 2, 2012 (the date we down- which will reflect heating and cooling needs. At the Naval War
loaded the data.) This span of data covers a temporal window College warm and cool air is not directly generated by any of the
before and after the three projects were implemented. three buildings, but energy is still consumed by the HVAC infra-
In addition, we downloaded weather data, specifically daily structure distributing this air. Further, it is not uncommon for
average temperature and rainfall, for the town of Newport, Rhode occupants to also use personal fans, heaters, coolers, humidifiers
Island from an online data center called Weather Underground.7 or dehumidifiers with variable frequency in response to tempera-
Weather data may be an important control in our regression ture changes. Second, X includes an indicator variable equal to one
analysis for two reasons. First, Project C specifically has sensors when school is in session and an indicator variable equal to one
related to sunlight, and we use rain as a proxy for scarce sunlight. when it is a weekend or federal holiday. These variables are
Second, because we have building level energy consumption data, intended to control for varying levels of human presence. Third,
this aggregate measure includes HVAC distribution, as well as X includes an indicator variable term for each hour of the day. This
private heating and cooling behavior if the building set point is captures normal variability in energy consumption experienced
undesirable. throughout the day.
We estimate Eq. (1) for temporal window sizes of 1, 2, 3 and 4
2.3. Empirical models months. We construct the windows such that equal portions of
observations fall before and after project implementation. Due to
Our empirical analysis seeks to estimate the impacts in terms of the intrinsic complexity of energy use patterns within buildings,
energy savings of the three energy conservation projects imple- reducing window size aids in minimizing variation that is other-
mented on the Naval War College campus. The building-level wise difficult to model. This is especially true for buildings where
energy consumption data downloaded from the modified electri- significant human presence, and thus human behavior, make up
cal meters for periods of time both preceding and following the for a substantial portion of energy consumed. For these reasons,
installation of each project are used to identify the respective we generally prefer small window sizes that allow us to estimate
impacts of those projects. project impacts while reducing the presence of variation attribu-
Since the energy conservation projects were implemented at a table to unmodeled behavior.
known point in time and the impacts of the projects are expected The second model we estimate is:
to be in full effect directly following implementation, we employ a 23
regression discontinuity design to estimate project impacts, with kw ht ¼ ∑ β1k  Iðhour ¼ kÞ  post_treatment t
k¼0
time as the forcing variable and the date of implementation
23
marking the discontinuity (Lee and Lemieux, 2010). Our metho-
þ ∑ β2k  Iðhour ¼ kÞ  post_treatment t
dology is similar to Bento et al. (2012), who assessed the impacts k¼0
of a temporal discontinuity in the high occupancy vehicle lane 23
access policy on interstate travel times. Our first RD specification is weekend_holidayt þ ∑ αk  Iðhour ¼ kÞ
k¼0
kw ht ¼ β  post_treatment t þ f ðtÞ þ X ′t δ þ εt ð1Þ
weekend_holidayt þ f ðtÞ þ X ′t δ þ εt ð3Þ
where kw ht is the 15-min kilowatt-hour reading and post_treatmentt
is a binary variable equal to one after the implementation. β is our This model allows the treatment effect to vary for each hour of
the day for both weekdays and weekends. β1k and β2k are the
coefficients of interest in this model and measure average kW h
6
While the estimated impacts for the library and walkway projects are near savings per 15 min interval due to the project for days of typical
identical, this is simply a coincidence. Each space is lit by the same style 4′ 32 W and reduced human presence. X t is the same as in Model 1. This
fluorescent lamps; the library saw lighting load reductions for 151 lamps, and the
hallway for 181 lamps. Additional predicted reductions came from sensor type and
heterogeneity enables us to examine if these projects designed for
use estimates. energy efficiency may additionally have benefits in terms of peak
7
Website is www.wunderground.com. load reductions. By pairing the hour of day impacts with the true
364 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370

variable costs of providing electricity during the day, we can dramatically in the case of hour of day fixed effects, and tends to
monetize the additional impacts of peak load reductions.8 decrease the size of the treatment coefficient.
While RD is ideal for measuring the energy impacts of Projects Columns 3 through 6 present models for time windows ranging
A and B, we evaluate Project C using a simple before–after from 1 to 4 months, each including all covariates. The results show
estimator. As a result of an elongated sensor adjustment period, large variability in the point estimates, ranging from  2.18 to
expectations of full impact directly following implementation are  11.29.9 However, five of the eight point estimates are in the
unrealistic, and use of a regression discontinuity framework to smaller range  4.53 to –  6.85, and each of these are statistically
explore project impacts is questionable. Using before–after esti- significant at the 1% level. We choose the 3 months window as
mation, we make use of a much larger time horizon and include preferred over the other windows since the point estimates are
more controls. We estimate treatment effects for Project C using fairly consistent between the cubic and quartic time trend, and
the model: from that we choose the specification with the cubic time trend as
preferred. Thus, our preferred estimate of the treatment effect is
kw ht ¼ β  post_treatment t þ X ′t δ þ Z ′t φ þ εt ð4Þ
 4.53, which is interpreted as building-level energy consumption
As in the previous models, we regress our 15 min energy declining by 4.53 kW h after the implementation of the project.
consumption readings on an indicator variable for treatment and This is a 9% reduction in the average energy use for this building.
a vector of controls. In addition to X, which is the same as in Eqs. Fig. 1 presents a graphical form of the discontinuity for Project A.
(1) and (3), we introduce a set of controls unique to the analysis of This graph was created by fitting a cubic curve to residual errors
Project C, labeled above as Z. Included in this new term is a cubic from a regression of kW h on all covariates except the time
function of daily precipitation. Because this project relies on polynomial.
daylight sensors, we use precipitation to control for the outdoor Fig. 2 visually presents results for the second model, Eq. (3),
light levels entering through the windows. Z also includes month which allows for heterogeneous treatment effects by hour of day
fixed effects to capture seasonal variation in energy use, as well as for both weekdays and weekends.10 We use our preferred speci-
seasonal variation in ambient light. fication of a 3 months window with a cubic time polynomial. The
Lastly, we additionally estimate a heterogeneous treatment figure plots estimated usage for every hour of the day before and
effects model for Project C: after implementation of the project, and the difference between
23
the two lines is the estimated savings. The results suggest that the
kw ht ¼ ∑ β1k  Iðhour ¼ kÞ  post_treatment t largest impacts occurred during evening and weekend hours,
k¼0 which is consistent with intuition. On weekdays during typical
23 working hours, human occupancy is at its highest and the book
þ ∑ β2k  Iðhour ¼ kÞ  post_treatment t
k¼0 stacks are accessed with the highest level of frequency. During
23 these hours, reductions in energy use are small or non-existent
weekend_holidayt þ ∑ αk  Iðhour ¼ kÞ because the human presence is forcing lights on. The greatest
k¼0
reductions occur throughout the night and on weekends and
weekend_holidayt þ X ′t δ þ Z ′t φ þ εt ð5Þ holidays where the sensors are triggered less frequently.
Fig. 2 additionally plots real-time energy prices in order to
gauge how the reductions associated with this project correspond
to peak load. The price curve was constructed by Joskow (2012) for
3. Results
July 7th, 2010 in New England and reveals the sheer magnitude
(a factor of 6) in which energy prices can deviate throughout the
In this section, we present our empirical estimates for the
day as a result of increasing marginal costs of production.
energy savings of the lighting equipment projects, including
Additionally, the energy use patterns for this building display
models that allow for heterogeneous impacts by time of day.
similar fluctuations through the course of the day, thus confirming
Finally, we compare our estimates of energy savings to the ex ante
intuition about peak demand. Matching the estimated reductions
engineering estimates, and compare both to the costs of the
to the cost curve, we can calculate the benefits of the project in
projects.
terms of the true cost of production. As Project A's largest
reductions tend to occur during off peak hours, accounting for
3.1. Ex post estimates of energy savings the variable cost of energy generation actually decreases the
monetary benefits of this project by 10% (see Table 5 and the
3.1.1. Project A discussion in Section 3.2).
Table 2 presents the main results for Project A using Model 1.
We run separate regressions for window sizes of 1 month, 2
months, 3 months and 4 months, with varied control suites. 9
In an attempt to combat the substantial volatility in the treatment estimates
Results are shown for regressions using cubic and quartic poly- from Table 2, we estimate similar models but additionally include lagged kW h in
nomials of time. Each coefficient is interpreted as the average the specification, similar to Chen and Whalley (2012). The motivation with this
kW h reduction per 15 min due to the implementation of the model is threefold. First, lagged kW h is likely to have an enormous effect on
current kW h due to the persistence of activities that occur in these campus
project. buildings (and any building or residence). Second, to the extent that our covariates
The first three columns of Table 2 use the 1 month time do not model variation in kW h well, including lagged kW h will substantially
window and explore the impact of including various covariates, improve the fit of the model and perhaps decrease the volatility of the coefficient
starting with none in Column 1, then adding a cubic of tempera- estimates. Third, the errors in Eq. (1) may suffer from serial autocorrelation that
clustering by day does not address. The results show that lag kW h is an extremely
ture and dummy variables for weekends, holidays, and whether
strong predictor of current kW h, however the total treatment effects are quite
school is in session in Column 2, and finally adding hour of day similar with the estimates in Table 2. Inclusion of lagged kW h had only a modest
fixed effects in Column 3. Adding covariates increases R-squared, impact on the volatility of the treatment effect. These results, as well as results
from models including lagged kW h for Projects B and C are available from the
authors by request.
8 10
Full accounting of social benefits of these energy efficiency projects would We choose not to present the results in table form as there are an unwieldy
take into account the social cost of power plant generation, including criteria 96 coefficient estimates that result from Eq. (3). Regression output used to
pollutants and carbon dioxide, but that is beyond the scope of this paper. construct these figures can be obtained from the authors.
 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370 365

Table 2
Regression discontinuity estimates of energy savings for Project A.

Time window

1 Month 2 Months 3 Months 4 Months

(1) (2) (3) (4) (5) (6)

Panel A: Third order polynomial

post_treatment  7.69  7.99  6.85  5.38  4.53  2.18
(4.48) (2.12)nnn (2.11)nnn (1.58)nnn (1.48)nnn (2.10)
R-squared 0.04 0.19 0.85 0.84 0.84 0.83

Panel B: Fourth order polynomial

post_treatment  5.73  4.38  3.12  11.29  6.26  5.69
(7.53) (2.27)n (1.96) (2.75)nnn (1.52)nnn (1.63)nnn
R-squared 0.11 0.19 0.85 0.84 0.84 0.83
Observations 2880 2880 2880 5760 8640 11459
Temperature cubic, weekend, holiday and school in session N Y Y Y Y Y
Hour of day fixed effects N N Y Y Y Y

Notes: Each coefficient comes from a separate regression of Eq. (1). Standard errors are shown in parentheses and are estimated using the Eicker–White formula to correct
for heteroskedasticity and are clustered at the day level. *, ** and *** indicate significance at 10%, 5% and 1%, respectively.

Project B, and in addition choose the cubic polynomial. Thus, our

5
preferred estimate of the treatment effect is  0.53, which is
interpreted as building-level energy consumption declining by
0.53 kW h after the implementation of the project. This is a 13%
0 reduction in the average energy use for this building. Fig. 3
Residuals (kWh)

presents a graphical form of the discontinuity for Project B. This

graph was created by fitting a cubic curve to residual errors from a
-5 regression of kW h on all covariates except the time polynomial.
Fig. 4 visually presents results for Eq. (3), which allows for
heterogeneous treatment effects by hour of day for both weekdays
-10 and weekends/holidays. We use our preferred specification of a 1
month window with a cubic time polynomial. The results suggest
that the largest impacts occurred during daytime hours. These
-15 results fit in with our intuition, given some previous knowledge of
the project building. In this small office, occupants reported having
-50 0 50
turned lights off lights when leaving the office at the end of the
Days Before & After Treatment
day manually before the installation of Project B. For this reason,
Fig. 1. Regression discontinuity diagram for Project A. Notes: Points on the graph we did not expect to see savings from the automated system
are daily averages of residuals from a regression of kW h on all covariates in Eq. (1) throughout the night. However, we do notice savings throughout
except the time polynomial. The curve is a third order polynomial interacted with
the day, which suggests that occupants were not turning off lights
an indicator variable for after treatment fit to the residuals.
while stepping away from their workspaces throughout the day,
and that the removal of fixtures had some impact.
3.1.2. Project B Comparing the heterogeneous treatment effects with the vari-
Table 3 presents the main results for Project B using Eq. (1). As able energy prices suggests that the benefits of this project
with Table 2, Columns 1 through 3 explore the impact of including increase as Project B's largest reductions tend to occur during
different control variables. Including a cubic of temperature and peak hours. Monetizing the peak load reductions causes estimated
the indicator variables for weekends, holidays, and school in benefits to increase by 14% over a flat electricity rate (see Table 5
session has a dramatic effect on the treatment effect coefficients, and the discussion in Section 3.2).
in one case changing the sign from positive to negative, in the
other case just making the coefficient more negative, and in both
cases making the coefficient statistically significant. This confirms 3.1.3. Project C
intuition that covariates are more important with Project B as this Table 4 presents the main results for Project C using Eq. (4).
is where there is the most human influence over energy con- Similar to Tables 2 and 3, the columns of Table 4 explore the
sumption. Including hour of day fixed effects increases R-squared impacts of including various covariates. In contrast to earlier
and decreases the size of the treatment coefficient, consistent with results, however, the coefficient estimates are remarkably stable.
the results from Project A. Excluding Column 1, which does not include any covariates,
Columns 3 through 6 present models for time windows ranging treatment effect estimates range from  1.65 to  1.51. Further,
from 1 to 4 months, each including all covariates. The eight all estimates are statistically significant at the 1% level. We choose
coefficients presented in those four models range from  0.74 to Column 5, the model with all covariates, as preferred. Thus, our
0.36. As the time window grows, the coefficients become less preferred estimate of the treatment effect is 1.51, which is
negative, even becoming positive in some cases, and lose statis- interpreted as building-level energy consumption declining by
tical significance. We interpret these findings as our empirical 1.51 kW h after the implementation of the project. This is a 7%
model being inadequate to control for human behavior (again reduction in the average energy use for this building.
because of human behavior) through changing seasons. Conse- Fig. 5 visually presents results for Eq. (5), which allows for
quently, we choose 1 month for our preferred time window for heterogeneous treatment effects by hour of day for both weekdays
366 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370

350 250

300
200

250
150

$/MWh
kWh
200

100
150

50
100

50 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Time of Day (Hourly)

350 250

300
200

250
150

$/MWh
kWh

200

100
150

50
100

50 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Time of Day (Hourly)

Fig. 2. Hourly kW h reductions for Project A. Notes: Plotted points for before and after treatment are derived from coefficient estimates of Eq. (3) using a 3 month window.
Coefficients are multiplied by four in order to get kW h on an hourly basis. Real-time energy prices for a summer's day in New England borrowed from Joskow (2012).

Table 3
Regression discontinuity estimates of energy savings for Project B.

Time window

1 Month 2 Months 3 Months 4 Months

(1) (2) (3) (4) (5) (6)

Panel A: Third order polynomial

post_treatment 0.32  0.80  0.53  0.21 0.36 0.22
(1.34) (0.32)nn (0.28)n (0.28) (0.28) (0.28)
R-squared 0.04 0.21 0.68 0.65 0.65 0.65

Panel B: Fourth order polynomial

post_treatment  0.36  0.94  0.74  0.23  0.20 0.25
(1.93) (0.39)nn (0.24)nnn (0.43) (0.30) (0.30)
R-squared 0.12 0.21 0.68 0.66 0.65 0.65
Observations 2880 2880 2880 5760 8640 11520
Temperature cubic, weekend, holiday and school in session N Y Y Y Y Y
Hour of day fixed effects N N Y Y Y Y

Notes: See notes to Table 2.

and weekends/holidays. We use our preferred specification which and weekends. These results are in line with intuition. We see the
includes temperature cubic, precipitation cubic, a school in session greatest savings during the brightest hours of the day, which
term, and monthly fixed effects. The results suggest that the suggests that the daylighting controls are effective at keeping
largest impacts occurred during daytime hours, both on weekdays lights off while the hallway is bright enough not to necessitate
 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370 367

them. However, we see few energy reductions throughout the Comparing the heterogeneous treatment effects with the vari-
night. Information we gathered on the influence of human behavior able energy prices suggests that the benefits of this project
for this project helps to explain this occurrence. The hallway is the increase as Project C's largest reductions tend to occur during
main point of access to several buildings on base, and it is regularly peak hours. Monetizing the peak load reductions causes estimated
frequented by security guards who perform rounds throughout the benefits to increase by 25% over a flat electricity rate (see Table 5
night. For this reason, lights are being triggered, and savings during and the discussion in the next section).
nighttime and early morning hours are minimal.

3.2. Comparing engineering to econometric estimates

0.5
Table 5 compares ex ante engineering estimates to ex post
econometric estimates of kW h and dollar savings for each project.
Empirical treatment effects are given for ‘Total’, which is derived
from Eqs. (1) and (4), and ‘Hourly’, which is derived from Eqs.
Residuals (kWh)

(3) and (5). kW h savings are converted into dollar savings using a
0 flat cost of electricity for all entries except the last in each group,
‘Hourly (variable energy prices)’, where Joskow's (2012) typical
daily cost curve is used.
For each of the three projects, we see that the empirical total
treatment effect exceeds the engineering estimate. By confirming
the engineering estimates, the empirical results offer support that
-0.5
implementation was cost-effective. For Project C, the engineering
and econometric estimates are very similar, however, for Projects
-20 -10 0 10 20
A and B, the econometric estimates are substantially larger
Days Before & After Treatment
than the engineering estimates. We know that the engineering
Fig. 3. Regression discontinuity diagram for Project B. Notes: See notes to Fig. 1. estimates were made to be conservative, but not erroneously so.

50 250

45

40 200

35

30 150

$/MWh
kWh

25

20 100

15

10 50

0 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Time of Day (Hourly)

50 250

45

40 200

35

30 150
$/MWh
kWh

25

20 100

15

10 50

0 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Time of Day (Hourly)

Fig. 4. Hourly kW h reductions for Project B. Notes: Plotted points for before and after treatment are derived from coefficient estimates of Eq. (3) using a 1 month window.
Coefficients are multiplied by four in order to get kW h on an hourly basis. Real-time energy prices for a summer's day in New England borrowed from Joskow (2012).
368 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370

Table 4
Before–after estimates of energy savings for Project C.

post_treatment (1) (2) (3) (4) (5)

R-squared  2.63 (0.31)nnn  1.65 (0.25)nnn  1.65 (0.25)nnn  1.51 (0.29)nnn  1.51 (0.29)nnn
0.03 0.17 0.17 0.24 0.40

Temperature cubic, weekend, holiday and school in session N Y Y Y Y

Precipitation cubic N N Y Y Y
Month FE N N N Y Y
Hour of day FE N N N N Y

Notes: All regressions come from an Eq. (1) specification. There are 61,627 observations. Standard errors are shown in parentheses and are estimated using the Eicker–White
formula to correct for heteroskedasticity and are clustered at the day level. *, ** and *** indicate significance at 10%, 5% and 1%, respectively.

120 250

110
200

100

150
90

$/MWh
kWh

80
100

70

50
60

50 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Time of Day (Hourly)

120 250

110
200

100

150
90
$/MWh
kWh

80
100

70

50
60

50 0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Time of Day (Hourly)

Fig. 5. Hourly kW h reductions for Project C. Notes: Plotted points for before and after treatment are derived from coefficient estimates of Eq. (5). Coefficients are multiplied
by four in order to get kW h on an hourly basis. Real-time energy prices for a summer's day in New England borrowed from Joskow (2012).

One possibility is that implementation of high profile projects led for Project A, as shown in Table 1, was achieved because simple,
to energy conserving behavioral spillovers in other parts of the inexpensive sensors switching several lights were employed in a
building. Given the psychology of the Navy and the fact that the space with very low human traffic. Similar savings were achieved
Secretary of the Navy proclaimed ambitious energy goals, these in Project C, despite more frequent foot traffic, because photo-
projects may have caused individuals to try and conserve energy sensors ensured that lights would remain off during daylight
in other ways. hours, regardless of human presence. We see a much longer
Of the three projects, Project A and C offered the highest payback in Project B, which had a lower sensor-to-lights-
returns on the initial investments, as estimated by both engineer- controlled ratio, in a higher traffic location, and was not able to
ing and empirical estimates of savings. The swift payback period incorporate photosensors. Also, as sensors were installed in small
 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370 369

Table 5 and Marron, 1992; Metcalf and Hassett, 1999) and may help
Result summaries. update conventional wisdom about disparities between these
types of estimates.
Project Estimating Treatment effect Estimated annual
method measurement savings We use 15 min interval energy consumption data, and the high
frequency nature of the data allows us to go beyond validating the
kW h Dollars engineering estimates and assess the extent of peak load reduc-
tions. Because the marginal costs of energy production are highest
(A) Library Engineering Total 42,357 $4,725
Empirical Total 158,865 $17,723
during typical working hours, energy reductions during these
Hourly (flat energy price) 158,531 $17,686 times offer additional social benefits. We estimate models that
Hourly (variable energy 158,531 $15,950 allow for heterogeneous treatment effects by hour of day for both
prices) weekdays and weekends and find substantial differences in energy
(B) Office Engineering Total 9,223 $1,029
reductions across time. Using cost of electricity production data,
Empirical Total 18,608 $2,076
Hourly (flat energy price) 18,977 $2,117 we were able to monetize the additional benefits from peak load
Hourly (variable energy 18,977 $2,419 reductions and found that two of our three projects caused peak
prices) load reductions that could be valued at up to 25% of annual
(C) Hallway Engineering Total 42,362 $4,726 savings. Consideration of such social benefits in project develop-
Empirical Total 52,910 $5,903
Hourly (flat energy price) 52,968 $5,909
ment would be a meaningful step in the right direction towards
Hourly (variable energy 52,968 $7,365 addressing important global energy and economic challenges.
prices) While our ability to access accurate, high-frequency recordings
of building-level energy consumption as well as ex ante cost
Notes: Empirically derived ‘Total’ savings come from our preferred specification of
benefit analysis data allowed us to meaningfully pursue advances
Eq. (1) detailed in the text for Projects A and B and Eq. (4) for Project C. Estimated
“Hourly” savings come from Eq. (3) for Projects A and B and Eq. (5) for Project C. in the current literature, our data did present us with limitations.
kW h savings are translated into dollar savings using a flat rate of 11.2 ¢/kW h for Separating project impacts from normal building-level variability
“Total” and “Hourly (flat energy price)” and the Joskow (2012) hourly prices for and human behavior presented a challenge. The regression dis-
“Hourly (variable energy prices)”. continuity framework is ideal to address unmeasured variables
related to human behavior, but some of the results were still
office rooms, electricians had to constantly relocate their tools and curious. Future studies could benefit from the collection of site-
supplies which proved tedious. level human behavior data such as work schedules, computer use
The engineering cost savings were calculated using a constant practices and some measure of personal reaction to the imple-
price of electricity because that is how the Naval War College pays. mentation of energy reduction projects within the workspace, the
However, if these efficiency investments can also reduce energy introduction of which could modify occupant behavior in either
use during peak demand hours, the projects offer additional social beneficial or destructive ways.
benefits and, in a larger sense, speak to whether efficiency and While this paper provides a rare comparison of engineering
peak load goals can be simultaneously achieved. First, comparing and economic estimates of energy savings, it is only a case study.
empirically estimated savings for Total and Hourly, we see that the However, due to the prominence of energy efficiency at the local,
kW h estimates are nearly identical. This is as it should be since state, and national level and the increasing availability and
the models are nearly the same. The important difference is that affordability of smart meters, we hope and expect that this type
the Hourly model allocates the Total savings to the correct hours of of ex post verification will become more common. Through a
the day. Second, we can compare the dollar savings from the wealth of experience, researchers can disentangle various factors
Hourly model with flat energy prices to the Hourly model with that drive realized benefits and the efficacy of energy efficiency
variable energy prices in order to monetize the hourly allocation of projects can increase. Further, methods of calculating ex ante
reductions. For Project A, savings in dollars decline by about 10% estimates of energy savings may be refined if through experience
when accounting for variable energy costs. However, Projects B it is found that some methods of evaluation lead to over- or under-
and C show increases in dollar savings of 14% and 25%, respec- estimates.
tively, suggesting these projects offer substantial additional social
benefits beyond the reduced kW h. The key to the daytime-
weekday energy reductions for these projects were daylight Acknowledgements
sensors coupled with abundant ambient light (for Project C) and
motion sensors in an environment where lights may be needed We graciously thank Dr. Marion Gold of the URI Outreach
only intermittently through the day (Project B). In both cases, but Center and Admiral Phil Wisecup of the Naval War College for
certainly in the case of Project C, these technologies could be organizing the initial energy study. We thank Shawn Bogdan and
employed elsewhere to achieve similar results. Captain Ken Wavell whose oversight made the project implemen-
tations possible. We also thank Emi Uchida, participants of the
2012 Northeast Workshop on Energy Policy and Environmental
4. Conclusions Economics, and two anonymous referees for valuable comments.
All remaining errors are our own.
This paper takes advantage of a unique opportunity to compare
ex ante engineering estimates to ex post econometric estimates of References
energy savings for three energy efficiency projects at the Naval
War College. Our econometric estimates suggest that each project Allcott, H., Greenstone, M., 2012. Is there an energy efficiency gap? Journal of
Economic Perspectives 26 (1), 3–28.
succeeds in reducing energy consumption in their respective
Anderson, S., Newell, R., 2004. Information programs for technology adoption: the
buildings, and the magnitudes of savings are actually higher than case of energy-efficiency audits. Resource and Energy Economics 26, 27–50.
their respective ex ante engineering estimates. By confirming the Bento, A., Kaffine, D., Roth, K., Zaragoza-Watkins, M. (2012). The Effects of
engineering estimates, the results contrast to the findings of other Regulation in the Presence of Multiple Unpriced Externalities: Evidence from
the Transportation Sector. Cornell University Working Paper.
authors who find that engineering estimates overstate realized Chen, Y., Whalley, A., 2012. Green infrastructure: the effects of urban rail transit on
savings (e.g., Dubin et al., 1986; Nadel and Keating, 1991; Joskow air quality. American Economic Journal: Economic Policy 4, 58–97.
370 C. Lang, M. Siler / Energy Policy 61 (2013) 360–370

Dubin, J., Miedema, A., McFadden, D., 1986. Price effects of energy-efficient Mabus, R.E., Jr. (2009, October 14). Remarks by the Honorable Ray Mabus [PDF
technologies: a study of residential demand for heating and cooling. Rand Transcript]. Retrieved December 19, 2012 from 〈http://www.navy.mil/navy
Journal of Economics 17, 310–325. data/people/secnav/Mabus/Speech/SECNAV%20Energy%20Forum%2014%20Oct
Gillingham, K., Newell, R., Palmer, K., 2009. Energy efficiency economics and policy. %2009%20Rel1.pdf〉.
Annual Review of Resource Economics 1 (1), 597–620. Metcalf, G., Hassett, K., 1999. Measuring the energy savings from home improve-
Granade, H.C., Creyts, J., Derkach, A., Farese, P., Nyquest, S., Ostrowski, K., 2009. ment investments: evidence from monthly billing data. Review of Economics
Developing a Holistic Implementation Strategy. In McKinsey & Company, and Statistics 81, 516–528.
Unlocking Energy Efficiency in the U.S. Economy (5) [PDF document]. Retrieved Nadel, S.M., Keating, K.M., 1991. Engineering Estimates vs. Impact Evaluation
December 19, 2012 from http://www.mckinsey.com/clientservice/electricpo- Results: How Do They Compare and Why?. American Council for an Energy-
wernaturalgas/downloads/US_energy_efficiency_full_report.pdf. Efficient Economy, Washington, DC.
Hence, K., 2013. Naval Station Looks to Wind to Cut Energy Costs. ecoRI News. Newsham, G.R., Birt, B.J., Rowlands, I.H., 2011. A comparison of four methods to
Retrieved February 2, 2013 from 〈http://www.ecori.org/front-page-journal/ evaluate the effect of a utility residential air-conditioner load control program
2013/2/4/naval-station-looks-to-wind-to-cut-energy-costs.html〉. on peak electricity use. Energy Policy 39 (10), 6376–6389.
Joskow, P.L., 2012. Creating a smarter U.S. electricity grid. Journal of Economic RGGI Inc. (2012, November). Regional Investment of RGGI CO2 Allowance Proceeds,
Perspectives 26 (1), 29–48. 2011 [PDF document]. Retrieved January 5, 2013 from 〈http://www.rggi.org/
Joskow, P., Marron, D., 1992. What does a negawatt really cost? Evidence from docs/Documents/2011-Investment-Report.pdf〉.
utility conservation programs. Energy Journal 13, 41–74. Wolak, F.A., 2011. Do residential customers respond to hourly prices? Evidence
Lee, D.S., Lemieux, T., 2010. Regression discontinuity designs in economics. Journal from a dynamic pricing experiment. American Economic Review 101 (3),
of Economic Literature 48 (2), 281–355. 83–87.