European Journal of Operational Research 156 (2004) 267–286

www.elsevier.com/locate/dsw

Case Study

Designing a performance measurement system: A case study

Clemens Lohman a, Leonard Fortuin b,*
, Marc Wouters c

a
TLO Holland Controls bv, Noordhoek 37, 3351 LD Papendrecht, The Netherlands
b
Faculty for Technology Management, Eindhoven University of Technology, Paviljoen F.18, P.O. Box 513,
NL 5600 MB Eindhoven, The Netherlands
c
School of Technology and Management, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands
Received 11 January 2002; accepted 26 November 2002

Abstract

Performance measurement (PM) by means of local performance indicators (PIs) is developing into performance
management at a company-wide scale. But how should PIs at various levels in the organization be incorporated into
one system that can help managers, working at levels that range from operational to strategic? How do we convince
potential users and obtain their support when starting to develop such a system? How can we aggregate PIs? How do we
present results? This paper addresses these and related questions. It is based on a case study carried out at the European
Operations department of Nike, a company producing and selling sportswear worldwide. The study resulted in a
prototype system that basically is a balanced scorecard tailored to the needs of the company. The empirical findings
differ in some ways from the literature on developing performance measurement systems (PMSs) in Operations. Dis-
cussing these differences provides new theoretical and practical insights. They relate to the role of parallel initiatives for
PM, the role of standardized metrics, the continuous improvement of PMSs, and the normalization and aggregation of
measures. Our findings suggest that developing PMSs should to a large extent be understood as a co-ordination effort
rather than a design effort. The lessons learned cannot have universal validity, but may be helpful in similar kinds of
initiatives.

2003 Elsevier B.V. All rights reserved.

Keywords: Performance measurement; Balanced scorecards; Supply chain management

1. Introduction the capabilities of their performance measurement

systems (PMSs) over the last years [14]. Perfor-
The ability to measure the performance of op- mance measurement (PM) in the context of a
erations can be seen as an important prerequisite supply chain becomes more important. The reason
for improvement, and companies have increased is obvious: companies start looking at ways to
improve operational performance through a better
integration of operations across subsequent eche-
* lons and separate functions in the value chain.
Corresponding author. Fax: +31-475-316-732.
E-mail addresses: [email protected] (C. Lohman), l.fortuin However, there are many obstacles to implement
@tm.tue.nl, [email protected] (L. Fortuin), m.j.f.wou- PMSs. Empirical studies about such initiatives are
[email protected] (M. Wouters). limited in the academic literature.

0377-2217/$ - see front matter
2003 Elsevier B.V. All rights reserved.
doi:10.1016/S0377-2217(02)00918-9
268 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

Several developments have created a need for The literature on PM in operations describes
companies to improve their supply chain man- several methods for developing PMSs. A charac-
agement. First, cross-functional co-operation teristic of many of these methods is the focus on
needs to be improved along the supply chain to developing performance metrics and a PMS based
offer shorter delivery times, more flexibility and on the firmÕs strategy and processes (see for ex-
faster introduction of new products (see, e.g. ample [30]). The literature also addresses the
[2,10,27]). Many companies are organized func- comparison of desired performance measures with
tionally, i.e. around subsequent stages of produc- existing measures (to identify which current mea-
tion, which makes it difficult to control the supply sures are kept, which existing measures are no
chain. Serving customers better requires synchro- longer relevant, and which ‘‘gaps’’ exists so new
nization of functions such as marketing, sales, measures are needed, [33]) and the periodic revi-
distribution, manufacturing, and purchasing. Sec- sion of PMSs once implemented [7]. However, the
ond, better synchronization is not only important literature does not provide a good understanding
across functional boundaries, but also across na- of how the process of developing a PMS is im-
tional boundaries. Spanning these boundaries has pacted by existing PMSs, both within and outside
especially occurred in Europe, where many com- the operations, at a more fundamental level. The
panies have moved from strong national organi- objective of this paper is to provide empirical re-
zations with local production, products and sults on improving PMSs to support supply chain
customers, to an organization where production management, using a case study methodology. A
has become more specialized and one factory comparison of these empirical results with the lit-
serves a specific part of the product range for the erature provides new theoretical insights. The
whole of Europe. Sales and marketing have be- findings are based on a case study within the Eu-
come partly centralized. This moves demand ropean Operations function of Nike. Case study
management, product allocation, marketing, and research has a small but consistent tradition in the
distribution to a European level. So there is a need operations management literature. Scudder and
to manage the supply chain on a European scale Hill [41] reviewed empirical research in operations
[1]. A third development is that streamlining of management published in the years 1986–1995 and
Operations across a chain of separate companies concluded that the amount of published empirical
has become more important because this creates work increased, both in absolute number and as a
opportunities to offer better service to end- percentage of the total number of articles pub-
consumers against lower costs for the supply chain lished. They found that case study and survey are
in its totality. However, information systems for the most widely used research designs, with a
costing and PM have generally not been very consistent mix of both designs and surveys being
helpful for managing operations, because such used more extensively compared to case studies.
systems were based on overly simplified models of The theoretical contribution of this paper is to
manufacturing activities and resource consump- show the limitations of a ‘‘green field’’ approach in
tion, which produced inaccurate cost data. More- the development of PMSs. The presence of existing
over, in many companies there was a lack of measures and parallel PM initiatives may quite
non-financial measures [21]. fundamentally change the development from a
Four terms will be used throughout the paper. ‘‘design approach’’ to a ‘‘coordination approach’’
A performance indicator (PI) is a variable that focused at aligning the supply chain operations
expresses quantitatively the effectiveness or effi- PMS with existing performance measures and
ciency or both, of a part of or a whole process, or parallel initiatives outside the operations function.
system, against a given norm or target [17]. PM is Our findings point to the central role of a shared
the activity of measuring performance using PIs. A set of standardized performance metrics as a tool
PMS is a system (software, databases, and proce- for achieving such coordination.
dures) to execute PM in a consistent and complete The paper has the following structure. In Sec-
way. A PI also is called ‘‘performance metric’’. tion 2 we look at the literature relevant to our
 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 269

subject. Most papers appear to deal with a ‘‘green

field’’ situation, whereas in practice there are al-
ways performance reports and indicators that have
to be incorporated. Next we sketch the back-
ground of our case study, i.e. supply chain PM, in
Section 3. Many companies that want to improve
their PMS have to face five problem areas, namely
(1) a decentralized, operational reporting history;
(2) deficient insight in the cohesion between met-
rics; (3) uncertainty about what to measure; (4)
poor communication between users and producers
of PI; and (5) a dispersed IT infrastructure. The
Fig. 1. Process control loops [29].
case study is the subject of Section 4. It briefly
describes the company, the approach followed,
and the system designed. It also presents the most
important lessons learned. The design can be on the control of an organization [29]. Two levels
characterized as a hierarchical system for PM, for of control can be seen. At the operational level, a
managers at various levels. Basically it is a bal- comparison of input and output values with pre-
anced scorecard tailored to the needs of the com- defined goals takes place. If there is a discrepancy
pany. For that reason it has six rather than the between the actual value of the PI and the desired
usual four areas of attention. The question: ‘‘What goal, knowledge about the behavior of the orga-
is scientifically new?’’ is addressed in Section 5. In nization is used to find an appropriate action, e.g.
this section we discuss the results obtained with modifying the process. This is the control function.
emphasis on elements with a value that reaches At the tactical or strategic level the control loop is
beyond the proper case study. The conclusions used to evaluate and adapt control level 1, by
relate to the role of parallel initiatives for PM, the changing goals if necessary. With these two con-
role of standardized metrics, the continuous im- trol loops, PM extracts the right process infor-
provement of PMSs, and the normalization and mation and provides goal information needed to
aggregation of measures. The findings provide evaluate performance (comparison) as well as
some new theoretical and practical insights in these goals (evaluation). ‘‘Right’’ process information
areas, and our results suggest several avenues for means that the information should be relevant for
further research. the level of control (strategic, tactical, or opera-
tional) and the companyÕs strategic objectives.
PM is based on the firmÕs strategy. It aims to
2. Literature overview support the implementation and monitoring of
strategic initiatives. The selection of performance
PM is an important topic both in the ope- measures and the setting of targets for these
rational research literature and the management measures are seen as concrete formulations of the
accounting literature. While traditional PMSs are firmÕs strategic choices. Both financial and non-
based on costing and accounting systems, mea- financial measures are needed to translate the
suring performance in Operations requires a more strategy into specific objectives that provide
balanced set of financial and non-financial mea- guidelines for operational action for middle and
sures at various points along the supply chain lower management. The actual results achieved for
[3,16–18]. the various measures reflect how well the firm
PM is an activity that managers perform in succeeds in achieving these strategic choices [13].
order to reach predefined goals that are derived Reviewing the ‘‘actuals’’ versus ‘‘planned’’ may
from the companyÕs strategic objectives. Fig. 1 il- lead to taking corrective actions in order to in-
lustrates this idea by taking a systems perspective crease the likelihood of achieving the goals. But
270 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

the results may also lead to challenging and ad- Table 1

justing these goals and strategic choices [35]. The Three relevant aspects of performance
‘‘balanced scorecard’’ concept attracted a lot of 1. Resources
Expenses (e.g. distribution costs, inven-
attention as a label to broaden PM initiatives: (1) tory-related costs, service costs)
to include a variety of financial and non-financial
Assets (e.g. inventory carrying costs)
measures from various perspectives, (2) to pay 2. Output
Financial (e.g. sales, profit, return on
attention to relationships between different mea- investment)
sures, and (3) to link PM explicitly to strategy
Time (e.g. customer response time, de-
livery lead time, on-time deliveries, fill
development [23–26,38,40]. rate)
PM is also based on the characteristics of a
Quality (e.g. reliability, shipping errors,
firmÕs operations, which need to be reflected in the customer complaints)
definitions of performance measures used. A per- 3. Flexibility
Volume flexibility (ability to respond to
formance measure is seen as a metric to quantify changes in demand)
the efficiency and effectiveness of operations [36].
Delivery flexibility (ability to respond
Several authors provide reviews of the literature quickly to tight delivery requests)
on PM in operations [4,11,36]. As Operations
Mix flexibility (ability to respond to
changes in the mix of products demand)
changes and becomes more central to the success
New product (and modified product)
of companies, performance measures need to be flexibility (ability to respond to demand
improved to support new operations practices. for new products)
Many traditional PMSs in Operations put a one-
sided emphasis on minimizing direct costs through
low material costs, high capacity utilization, and Table 2
Nine steps to develop a PMS [36]
high direct labor efficiency. Modern manufactur-
ing systems and service operations, however, need Step Action
also clear measures on quality, throughput times, 1 Clearly define the firmÕs mission statement
flexibility, etc. [6,15,19,22,31]. We refer to [8] for 2 Identify the firmÕs strategic objectives using the
mission statement as a guide (profitability, market
an overview of performance measures used in share, quality, cost, flexibility, dependability, and
Operations. There, relevant aspects of perfor- innovation)
mance are resources, output and flexibility. Table 3 Develop an understanding of each functional areaÕs
1 gives a summary. role in achieving the various strategic objectives
The development of a PMS may conceptually 4 For each functional area, develop global perfor-
mance measures capable of defining the firmÕs overall
be separated into phases of design, implemen- competitive position to top management
tation, and use [7]. The design phase is about 5 Communicate strategic objectives and performance
identifying key objectives and designing measures. goals to lower levels in the organization. Establish
In the implementation phase, systems and proce- more specific performance criteria at each level
dures are put in place to collect and process the 6 Assure consistency with strategic objectives among
the performance criteria used at each level
data that enable the measurements to be made 7 Assure the compatibility of performance measures
regularly. In the use phase, managers review the used in all functional areas
measurement results to assess whether operations 8 Use the PMS
are efficient and effective, and the strategy is suc- 9 Periodically re-evaluate the appropriateness of the
cessfully implemented. This may also lead to established PMS in view of the current competitive
environment
challenging the strategic assumptions. The design,
implementation, and use of a set of performance
measures are not a one-time effort: a firm should
install processes that ensure continuous review of change. A typical development process is described
the system [5,7,33]. Review processes imply that a in [36]; see Table 2.
measure may be deleted or replaced, the target The process is often iterative, whereby measures
may change, and the definition of measures may are developed and adjusted as more information
 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 271

about strategy, customers, processes, etc. becomes typical steps such as defining a firmÕs manufac-
available. The appropriate measures are derived turing strategy through competitive priorities,
from such information in several rounds to review linking success factors to the competitive priori-
and revise the measures. The availability of data is ties, defining measures, implementation of mea-
one of the considerations in the design process. sures, and executing periodic maintenance. The
There is also much attention for updating perfor- ‘‘audit’’ step may lead to eliminating some existing
mance measures once they have been imple- measures and identifying gaps in the current
mented. Kaplan and Norton [24] emphasize using measurement system. In this paper, however, we
documents, interviews, and executive workshops describe an empirical study that shows how the
for gathering information and building consensus. presence of existing measures and parallel PM
Various approaches can be used to design the PMS initiatives may have more fundamental impact on
([29] based on [12]): the development of PMSs. Rather than creating
1. Asking: Techniques to find out the require- the need for an additional step to verify and ad-
ments of managers, such as interviews, group dis- just, it may change the nature of the development
cussions, planning meetings, and surveys, are most challenge from (a) designing a PM ‘‘as if’’ from a
often used to develop a PMS. clean sheet of paper to (b) making sure that an
2. Prototyping: Instead of focusing primarily improved Operations PMS is aligned with existing
on a thorough analysis of the information needed, performance measures and parallel initiatives
an initial set of requirements is specified and a outside the operations function. So far, this has
prototype system is built. Through interaction received little attention in the literature on PM in
with users of the system (managers), require- Operations.
ments are added or changed until the user is sat- In summary, PM yields a fundamental type of
isfied. management information needed for controlling
3. Planning methods: Methods that design ap- operations. It creates focus, triggers corrective
propriate measures based on the characteristics of action, is the basis for evaluating performance,
the firm, such as strategy, processes, and custom- and may help challenging and improving strategic
ers. For example, a method could be followed to choices. Both the management accounting litera-
determine a few areas (critical success factors) that ture and the operations literature focus on the
dictate the success of the firm. For such areas connections between strategy and PM––the role of
critical success factors are described, which leads PM as translating strategy into concrete goals and
to the definition of measures that capture these monitoring the delivery of strategy––and between
factors. Operations and PM––measures need to capture
4. Existing reports: Often a useful source of the relevant characteristics of the underlying op-
information to be used to design the PMS. erational processes. Approaches for developing
Implicit in many approaches for designing PMSs use various ways to gather information, and
performance measures is a ‘‘green field approach’’ there is much attention for an iterative process in
that does not pay explicit consideration to existing which measures are developed and adjusted as
measures. However, in many settings it is realistic more information becomes available about strat-
to acknowledge that reports relevant for managing egy, customers, processes, and the availability of
operations already exist at various levels in the data. There is also much attention for updating
organization, within and outside the operations performance measures once they have been im-
function. Medori and Steeple [33] is one of the few plemented. However, there is far less literature that
papers that explicitly discuss the relationship with provides an understanding of how the process of
existing measurement systems; their design method developing a PMS is impacted by existing mea-
includes an ‘‘audit’’ step to compare the newly sures (or new measures that are being developed
defined desired performance measures with the simultaneously as a result of other initiatives) at
measures that already exist. However, this is various levels both within and outside the opera-
merely one element of a process that includes tions function.
272 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

3. Supply chain performance measurement cohesion. This makes it hard to focus attention in
an effective way and causes a lack of overview.
Many companies are trying to improve supply This not only makes management to feel discom-
chain PM, in order to support managing opera- fort, but it also can lead to missing opportunities.
tions across supply chains. It may be useful to 3. Uncertainty what to measure––Often uncer-
think first of two extremes: (1) several functional tainty exists about what exactly should be mea-
or regional departments are each responsible for sured on a supply chain level. Since current reports
one aspect or one part of the supply chain and mainly cover parts of the supply chain, it is likely
only top management is responsible for the total that certain high-level metrics are lacking. This
financial results; and (2) a situation in which a adds to the managerÕs discomfort.
management team is responsible for the overall 4. Poor communication between reporters and
performance of the whole supply chain. Most users––Communication between the creators and
companies are somewhere in between. But as users of reports is often poor. The creators often
companies move towards a more integrated Op- hardly know their audience and the exact purpose
erations Management function across the supply of the reports. This results in poor readability and
chain, it becomes necessary to measure the per- limited usefulness. The users on the other hand
formance of the various parts of the supply chain sometimes do not know why they receive a certain
on various dimensions, in a consistent way. There report and so they do not use it at all. The lack of
is a need to define and measure performance for interaction make the reports outdated in relation
the supply chain as a whole and to be able to to the business as well as user preferences.
drilldown to different measures and different levels 5. Dispersed IT infrastructure––Companies use
of detail, in order to understand the causes of many information systems that are linked in some
significant deviations of actual performance from way. The dispersed IT infrastructure produces a
planned performance. However, many companies number of issues. Firstly, it adds to the lack of
seem to be facing serious difficulties in imple- data integrity between the reports. Since consid-
menting such supply chain-wide PMSs that cap- erable overlap exists between the systems, certain
ture various dimensions of performance at various data can be extracted from multiple sources and
levels in a consistent way. These difficulties have this often leads to inconsistency. Secondly, the
various causes: infrastructure does not provide visibility over the
1. Decentralized, operational reporting history–– supply chain, owing to the absence of connectivity.
There is often a history of decentralized reporting Thirdly, certain systems are not designed for re-
with a focus on local operational use within facto- porting uses or cannot provide data at reasonable
ries, transportation linkages, distribution centers, cost at all.
sales offices, etc. This has led to an uncontrolled These five complexes of difficulties raise the
growth of reports with many inconsistencies. These question how supply chain PM can be improved.
inconsistencies have to do with definitions of per- The objective of this paper is to report on an
formance metrics, sources of data for obtaining empirical study on the development of PM. By
measures, and ways of presenting reports. Man- doing so reflection on current theories becomes
agers who try to construct a total picture of the sensible.
supply chain from these reports find themselves
confronted with a large volume of (inconsistent)
information in a format that does not support 4. The case study
integrated analysis.
2. Deficient insight in cohesion between metrics–– The study took place at European Operations
Since current reporting has an operational focus, of Nike. The company was continuously improv-
the metrics are used to monitor sub-processes of ing its supply chain management. As part of these
the supply chain. These pieces of information are efforts management decided to assign some of
analyzed on an individual basis rather than in their resources for improvement projects to PM.
 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 273

Especially the integration of various local PI into a rope in 1977. Nowadays, there is one European
company-wide consistent system for PM was re- distribution center, the Customer Service Center
quired. The authors were involved through a de- (CSC) in Laakdal, Belgium. In 1999, Customer
sign project carried out as part of a university Service––responsible for order and query man-
program in international logistics. One author agement––also was concentrated in one place, the
carried out the project as a postgraduate student European Headquarters (EHQ) in Hilversum, The
to complete this program; the other two authors Netherlands.
coached and supervised the work. The project was The company is organized around three lines of
done at the company full-time during six months business: footwear, apparel, and equipment.
and can be described as action research [39]. The Footwear and apparel make up the largest part of
empirical observations and experiences are com- the business; they are almost equal in size.
pared to previous theory regarding the develop- Equipment is relatively small (5% of revenue),
ment of PMSs in order to develop a better but it is growing fast. The lines of business are
understanding of how such theories or concepts divided into product lines, and each line is divided
might work in real situations. This might be con- into categories. The total assortment per line of
sidered as a descriptive case study, or maybe an business can be characterized as ‘‘large’’. This
exploratory case study [32]. A group of about 10 holds in particular for apparel: it comprises of
company managers reviewed the results every few 60,000 stock keeping units in the supply chain.
weeks, and in-between such review meetings, there Comparable figures for footwear and equipment
were interviews with many more company man- are 25,000 and 1000, respectively. The product life
agers and frequent informal discussions with the cycles are short, which is normal in the clothing
company coaches and the director of Operations. industry. Most products are specially designed for
This paper reports on the situation at the end of a specific season. This holds less for equipment
the project. The company has continued the de- and not for basic products like socks, white shirts,
velopment and implementation of the PMS to etc.
modify the clustering of metrics, to increase data The European region comprises Europe, the
availability so that more metrics can actually be Middle East, and Africa (EMEA). The business
measured and reported reliably, and to change the volume in the last two areas is very small as
structure and presentation of the scorecards. compared to the European volume. They can,
First we will introduce the company in Section however, be considered as emerging, high potential
4.1. The design of the PMS is described in Section areas. The ‘‘big five’’ in Europe (UK, France,
4.2, while the development approach and the Germany, Spain, and Italy) make up 70% of the
critical choices made in this process are discussed total revenue of Nike Europe.
in Section 4.3. Next we discuss the companyÕs ex- Uncertainty of demand is an important char-
periences during the first year of implementing and acteristic. Although market intelligence is widely
using the PMS, in Section 4.4. In Section 4.5 we present and aggressive marketing is being used,
discuss the method and how our empirical findings consumer behavior is hard to predict when it
suggest that elements previously not pointed out in comes to fashion. Although only a small portion
the literature may be critical for the development of products is delivered directly to consumers, via
and implementation of a PMS in Operations. To the Internet (www.Nike.com) and Nike retail, their
conclude, Section 4.6 lists some practical sugges- buying volume does affect the sales of Nike to
tions derived from this project. retail organizations.
This paper does not deal with the Nike supply
4.1. The company chain in its totality: it covers the demand for
the European market (sourced worldwide) and
Nike is active in the clothing and sport acces- is restricted to Operations, which consists of
sory industry. After becoming successful in the Transportation, Warehousing, and Customer Ser-
USA, Nike started exporting its products to Eu- vice. By putting together various requests from
274 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

management, the project objectives became as the metric selection. It resembles the balanced
follows: scorecard, but it is extended with a cluster for
Sustainability and one for People. This extension
1. to develop a set of high-level performance met- is made in order to fit OperationsÕ specific char-
rics tailored to the specific business needs for acteristics and to pay explicit attention to these
use by the senior supply chain management areas. Fig. 2 depicts the clusters together with the
team, i.e. Operations, while including existing questions that should be answered by the metrics
local metrics as much as possible and sensible; included. Clearly, mission and strategy are the
2. to design a format, i.e. a scorecard, displaying starting point and source for objectives in the six
the metric scores at the level of Nike as well clusters. All relevant areas for Nike Operations are
as that of the business units. represented. Here are some details:
Customer––Nike Operations is connected to its
4.2. The design customers by means of a physical process (the
delivery of products) and an informational process
Our case study resulted in a new prototype (via CSRÕs, i.e. Customer Service Representatives).
system for PM. Its scope is limited to Nike Europe By using information directly obtained from the
and the Operations function, including Ware- customers as well as information about the pro-
housing, Transportation and Customer Service. cesses on the interface between Nike and its cus-
Point of departure is a set of design guidelines that tomers, the performance towards the customers is
are tailored to NikeÕs characteristics. Application measured on aspects such as customer satisfaction,
of these guidelines produced a set of performance shipment queries, and order fill rate.
metrics, and a scorecard for displaying the corres- Sustainability––This cluster contains metrics
ponding information. that relate to the interaction between Nike and its
environment. In the recent past, the company has
4.2.1. Performance metrics started several projects to increase the awareness
The metric selection should contain output- for sustainable growth.
related PIs as well as (leading) operational indi- Financial––The Financial cluster offers a view
cators. Following this guideline of the design on OperationsÕ contribution to shareholder value
approach, we developed a clustering method for by looking at costs and revenue and margin in-

Fig. 2. Metric clustering for Nike Operations.

 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 275

fluencing factors. Since Operations is a cost center, button on the bottom allows updating of the
it can contribute to shareholder value by realizing gauges according to changes in the data. If a user
low costs and facilitating the generation of reve- wants more information about the performance of
nues. The financial aspect includes cost per unit a cluster, he can click on the button below the
measures and inventory-related measures. Abso- corresponding gauge to enter a lower information
lute costs are not included since other mechanisms layer.
are present to monitor cost versus budget. Mid level––The next level in the scorecard
Process Improvement––The Process Improve- shows the highest-level indicators for the selected
ment cluster contains metrics that relate to long- cluster. Fig. 3B shows this for the cluster called
term improvement trajectories and strategic issues, Customer. The overall cluster score is repeated in
such as the progress of key projects, the quality of the black box. The score of each key PI is depicted
supply, and the complexity of Operations. The numerically and graphically by the bar chart. The
score of this cluster needs to be sufficient in order upper bar indicates the score of the current month;
to safeguard growth in the future. the lower bar that of the previous month. The user
Product Flow––The Product Flow cluster con- can find comments on the scores by using the pull-
tains metrics that track effectiveness in the supply down list underneath the black box. Once again,
chain. Basically, the performance of this cluster definitions can be found by clicking the button in
forms the basis of the performance of the delivery- the right corner on the bottom. The ‘‘back’’ button
related metrics in the Customer cluster. Accuracy in the upper right corner returns the user to the
and throughput of good flows at the subsequent dashboard. Clicking the buttons on the right of the
stages of the supply chain are the focus in this bar chart saying ‘‘details’’ takes the user a level
cluster. down in the hierarchy. Depending on the position
People––The organizational health can be as- in the hierarchy another mid-level screen appears
sessed by means of this cluster, such as employee or a lowest-level screen is reached.
satisfaction, professional development, and diver- Lowest level––This level provides the user with a
sity. presentation of the performance of an indicator
that is tailored to its characteristics. An important
4.2.2. Scorecards common characteristic for the graphs on this level
A structure with three layers is used for dis- is that they show the development over time. In
playing the information. The characteristics per Fig. 3C we see the development of ‘‘CPU Total’’
layer are described below. The scorecard prototype on the scorecard issued in the beginning of Jan-
is built in Microsoft Excel with supplementary uary 2001 covering performance up to December
programming in Visual Basic. 2000. The monthly actual values are displayed
Top level––If a user opens the scorecard file, the together with a 12-month moving average. The
screen will show the highest level of the PI struc- latter cancels out seasonal fluctuations and facili-
ture: the metric clusters (see Fig. 3A). We call it the tates trend analysis. The seasons are depicted
‘‘dashboard’’ as, like in a car, it displays high-level, below the graph as a reference. The figure shows
aggregated performance information. The gauges the CPU for the total business; the user can choose
depict the score of each cluster numerically and to view CPU for individual business units by
graphically. The main pointer indicates the score clicking one of the buttons in the upper right box.
of the current month, a shadow pointer that of the The way of displaying performance used here is
previous month. In this way the user can see how specific for this indicator. For other metrics other
the metric value develops. The color of the cluster graphical formats are used if more appropriate.
names on the buttons underneath the gauges in- Metrics and scorecards are the key elements of
dicates whether the underlying metrics are out of our system. Therefore, they have been discussed in
their control range. The buttons on the left side of some detail. Other aspects, however, are important
the dashboard give access to a user guide and to a as well: normalization and aggregation, usage and
list of metric definitions used in the scorecard. The maintenance.
 276
C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

Fig. 3. Scorecards at three levels. (A) Top level; (B) mid level; (C) lowest level.
 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 277

4.2.3. Normalization and aggregation director Operations, the functional directors

The normalization method proposed is based (Transportation, Warehousing, and Customer
on a linear 0–10 scale, the usual range for school Service), and the three business unit Operations
marks. It appeals to oneÕs imagination and makes directors. They will use the scorecard on a monthly
readability and interpretation of actual metric basis to facilitate review of the organizationÕs
values easy. Two steps need to be taken for nor- performance. Furthermore, the General Manager
malizing the metric scores: Nike Europe can use the scorecard to facilitate a
quarterly review of Operations. Distributing the
1. Set performance targets––The target is the start- scorecard to all OperationsÕ managers facilitates
ing point for defining the metric score range communication and alignment of behavior to-
that corresponds with the 0–10 scale. wards the organizational objectives.
2. Normalize scores to a 0–10 scale––A target will The scorecard and its contents (i.e. the metrics)
lie somewhere between 0 and 10. Since consis- cannot be considered as static entities: they must
tency is recommended when using a normalized be maintained and updated to remain relevant and
scale, the values 0–10 should always have a useful for the organization. Two events can be
same meaning, regardless the metric observed. identified that trigger changes in the scorecard and
In our system the score 8 corresponds to the tar- the metric selection [9]:
get. This means that if the target is hit, the met-
ric gets a score of 8 or higher. For practical • The scorecard does not anymore support the
reasons we include a lower and an upper bound control of (a part of) the business––During per-
on the scale. formance review sessions it can appear that
business areas or current or new challenges
Fig. 4 gives an example. Suppose the target for are not covered in the scorecard. Then addi-
delivery performance is 90%. We let this corres- tional requirements are formulated for the next
pond to a score of 8. Delivery performance can edition of the scorecard.
vary theoretically between 0% and 100%. Letting • The organizational objectives change––Since
100% be the upper bound means that on a linear performance metrics are aimed at tracking the
0–10 scale 10% lies above 8 (100  90). This makes performance towards the organizational objec-
50% (½10=ð10  8Þ  10%) the lower bound of this tives, a change in strategy hits the heart of the
metric. Aggregation means nothing more than scorecard.
calculating an average of the normalized scores.
This can be a weighted average or not. Manage- The complete design process has to be repeated
ment needs to decide on the relative importance of taking into account the new business conditions.
the metrics for the aggregation process. Given these two triggers, we propose a mechanism
for maintenance and an updating:
4.2.4. Usage and maintenance
The scorecard is made primarily for the senior • Monthly scorecard reviews during or after
supply chain management team, consisting of the the performance review sessions––During the
monthly review performance sessions, the
scorecard owner has to be present. He carefully
evaluates the use of the tool and its contents,
and gathers additional requirements to be in-
corporated the next month.
• Yearly redesign of the scorecard and its con-
tents when launching new business plans––Once
a year, the complete scorecard and correspond-
ing metric selection should be updated in accor-
Fig. 4. Example of normalization. dance to the new business plans. This means
278 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

that the design approach has to be followed • Headquarters was taking initiatives to develop
each year to assure an up-to-date PM tool. a worldwide scorecard for the Operations func-
tions. Zooming-in on Europe should provide
4.3. The development approach information that would be consistent with the
scorecards that European Operations would
In view of the literature on PM, rapid proto- use themselves.
typing seemed a logic start. The idea was to look at
current reports and add some potentially relevant Furthermore, collaboration with the business ana-
measures to produce a first version of a report. lysts who produced current reports appeared more
Future users then would review this, i.e. the time consuming than anticipated. The prototyping
management team that was responsible for the approach works if gradually more of the proposed
supply chain. A prototype could give concrete metrics are presented on the basis of real data and
examples of how the new PMS would look and actual measurements. But many people placed
this could stimulate discussion among the users demands on the business analystsÕ time. Producing
and generate feedback to the designer. Next, by credible numbers instead of merely ‘‘theoretical’’
using the comments of the users, improvements exercises took a lot of time.
would become possible, until eventually a satis- Hence, a new approach was chosen, in close
factory system is obtained that can be made collaboration with management at the European
available to users at various levels and locations in Headquarters. Central to the new approach be-
the organization. The prototyping would also be came the development of a metrics dictionary with
used to experiment with software designed for some 100 metrics. The discussions stepped away
producing reports. However, it soon became ap- from the presentation and structure of scorecards.
parent that there were reasons for taking a differ- It focused on getting a detailed understanding of
ent approach. the individual metrics that were used as part of the
First of all the design efforts had to be aligned various ongoing initiatives described above. At the
with other performance reporting initiatives. Nike start of the project metrics were often documented
had already functional scorecards and periodic in an ambiguous way causing failures in commu-
reports (about 140 in total), also outside Opera- nication between reporting employees and man-
tions. Here are some details: agers. In order to increase the quality of the
metrics and communication about the information
• Many reports existed at lower operational levels they list, a template was developed. We followed
and when possible the existing metrics would be Neely et al. [37] and added some more attributes
used, which required a careful understanding of (see Table 3). In other words: more work was done
these metrics. and more time spent before first results were
• The various functions within Operations (Ware- shown to potential users. This process of using the
housing, Transportation, Customer Service) new template and a metrics dictionary caused vivid
had developed functional scorecards, and these discussions about what had been reported in the
would form the main experience base for de- past and how this should be in the future. Fur-
signing the supply chain-level scorecards. In a thermore, template and dictionary made all re-
way, the supply chain-level scorecard would porting employees more aware of PM, which can
be a combination of the functional scorecards be seen as a positive side effect.
enriched with measures that would be pertinent In this new approach, reporting has become a
at the supply chain-level. process of continuous improvement, but in a dif-
• Outside Operations other functions were devel- ferentiated way. On the one hand, the metrics
oping measures that could be incorporated in dictionary has been fixed. This is the central ele-
the supply chain scorecard, such as measures ment to ensure co-ordination between the various
from the Human Resource function about efforts that are going on and being developed over
training and retention. time. On the other hand, the structure of reporting
 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 279

Table 3
Metric definition template: attributes with their explanation (based on [37] with elements added)
Metric attribute Explanation
Name Use exact names to avoid ambiguity
Objective/purpose The relation of the metric with the organizational objectives must be clear
Scope States the areas of business or parts of the organization that are included
Target Benchmarks must be determined in order to monitor progress
Equation The exact calculation of the metric must be known
Units of measure What is/are the unit(s) used
Frequency The frequency of recording and reporting of the metric
Data source The exact data sources involved in calculating a metric value
Owner The responsible person for collecting data and reporting the metric
Drivers Factors that influence the performance, i.e. organizational units, events, etc.
Comments Outstanding issues regarding the metric

of metrics: it is impossible for a manager to make

decisions on the basis of 100 unstructured metrics.
Clustering the metrics into six perspectives such
that each cluster is connected to a coherent set of
organizational objectives made a first step. This
has much similarity with the balanced scorecard
[23] that prescribes the use of four standard per-
spectives. Our six clusters were defined via inter-
Fig. 5. Hierarchy of metrics. views with individual managers and a workshop
with the group of managers in Operations that
were involved in this project on a regular basis, as
can and should be changed much more frequently mentioned in the introduction to Section 4. The
(almost every month in the beginning, becoming important point of our method is that this clus-
more stable after a number or months). The report tering helped in creating a clearer connection
uses clusters of metrics and hierarchical levels (Fig. between metrics and strategy, and it improved
5). The selection of metrics, the scope of the clus- communication about the metrics. A second deci-
ters, the structure of the hierarchical levels, and the sion to manage complexity was to use a hierar-
presentation of the reports are all aspects that chical structure within each cluster, thereby using
needed to be reviewed frequently, parallel to using so-called ‘‘engineered indicators’’ i.e. metrics based
the actual metrics for managing the supply chain. on two or more lower-level metrics. The process of
In short, the approach has developed into (1) the constructing the hierarchy increased insight into
creation of a metrics dictionary, combined with (2) the cohesion between metrics. It can be seen as a
an overview of initiatives, leading to (3) a draft start for exploring means–end relationships among
version of the scorecards and (4) explicit proce- the metrics.
dures for using and updating the metrics dictio- Other decisions concerned the organization of
nary. the development process. Bringing together all
Several choices were important during the de- people working on parallel initiatives in the field of
sign process. One such choice that we mentioned PM within the organization was the first step to
above was to focus on defining fixed metrics first make. By establishing a cross-functional alignment
while being much more flexible regarding the forum, relevant knowledge was concentrated, in-
scorecards (selection, level of detail, presentation consistencies were quickly located and communi-
of metrics). Other choices had to do with reduction cation was institutionalized. The forum sessions
of the complexity of working with a large number were mainly devoted to discussions about metric
280 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

definitions and reporting formats, and to ex- concluded later, the power of this tool for dis-
changing ideas about developing and using PM in playing scorecards in many sophisticated ways
the organization. The results of these sessions were distracted the attention from the underlying met-
documented in the metric dictionary. These dis- rics. It also tended to lead to high expectations and
cussions, thus, became the focal point of building a lot of debate about the displaying format. An-
an understanding of the existing measures and other disadvantage was that the tool would be-
reports, and of parallel initiatives to develop a new come expensive if it had to be made available for
PMS. The new PMS was synchronized with ex- many users of reports. Such a wide introduction of
isting procedures and performance reports mainly new software in the company involved much more
by entering detailed definitions of existing and complicated decision-making processes. Experi-
proposed performance measures in the metric menting with the tool caused delays and took too
dictionary. We expected that such synchronization much time from the researchers. After a couple of
would be complex. It turned out to be not so dif- months it was decided to take a step back and only
ficult, because measures in the existing reports al- use a generally available spreadsheet tool, MS
though presented in different ways, used closely Excel.
related definitions, or it was easy to agree on a
common definition. In Section 4.4 there is more 4.4. Experiences
information on how this process of coordinating
existing and new measures continued after the The company has continued developing and
development projected reported here. implementing the PMS after the initial design
Next to the forum, managers, data specialists project described in this paper. They have experi-
and ICT staff were consulted regularly during all mented with different ways of clustering the met-
stages of the development process. Especially rics and presenting the scorecards. They have also
management, being the ultimate user, appeared to gone further in gathering the data required for
need a sense of ownership in order to use the re- actually measuring and reporting all metrics in a
sulting product. It was interesting to see that, when reliable way. The following observations sup-
using dummy data, managers seemed less moti- port the notion that the development of PMSs in
vated to contribute to the development process Operations is a continuous and experimental pro-
than when using real data. In order to get their full cess.
attention, we decided to discuss only those metrics
for which at least one real figure could be shown. 4.4.1. The role of a PM manager
The data specialists and ICT staff were necessary One person of the cross-functional initiatives
to develop metrics that can be reported at rea- group manages the performance reporting process.
sonable cost. In most cases, the employees able to He collects data from various sources, publishes
deliver the most reliable data (including interpre- the monthly reports, discusses the results with
tation) about a process are directly involved in that managers and manages the agenda of follow-up
process. Apart from the fact that they are indi- actions, and continues to develop the PM method.
rectly asked to deliver information about their The scorecard is reported every month. In the
own performance, often no time is explicitly bud- second or third week of the month the functional
geted for this activity. Co-operation with these scorecards (transportation, warehouse, customer
employees appeared to be cumbersome and man- service) and a so-called ‘‘Operations Flash’’ are
agement needed to communicate the priority of completed. In the following week the ‘‘Operations
the project several times in order to improve this. Flash’’ is discussed in one-on-one review meetings
The last category of decisions concerned the between the managers who are responsible for the
tools used. Early in the project a start was made functional scorecards and the manager of the
with the use of a software tool that helps to cal- performance reporting process, whereby the func-
culate and visualize the performance on a hierar- tional scorecards are used as background infor-
chical set of metrics. However, as could be mation. In the third or fourth week, always on a
 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 281

Monday, the ‘‘Operations Flash’’ is discussed in tions. This led to the development of the Opera-
the Operations Staff meeting. The next day tions Flash report, a one-page report showing the
(Tuesday) the Operations Flash report and the metrics with a focus both on business unit and on
action items that have been taken up in the Op- countries. As expected, the use of the PMS trig-
erations Staff meeting are discussed with world gered a process of continuously revising the format
headquarters in the monthly telephone conference of presenting the scorecards.
with the regional management team (the so-called
regional logistics call). Every quarter a global 4.4.3. Coordination of initiatives
scorecard is discussed more extensively in the The topic of coordination between various PM
quarterly global supply chain leadership meeting. initiatives remains to be a central element of de-
The global Operations scorecard shows a set of veloping and implementing PMSs. This has be-
measures for four different regions (EMEA, US, come apparent for the implementation of a global
Asia Pacific, Canada). The PMS has become a tool Operations scorecard. Some measures that had
that is actually being used in various meetings and been developed in this project (i.e. for the EMEA
one-on-one situations to guide action and to region) have been adopted without modification,
monitor results. The experiences prove the im- while some other measures in the EMEA region
portance of someone responsible for the whole have been adjusted. Metrics owners, who are re-
process, not just as a reporter, but as a manager in sponsible for improving the performance measures
charge of concrete follow-ups and monitoring the that are part of their functional scorecards,
effects of actions, as well as being responsible for sometimes want to change the definition of certain
improving the PMS itself. This role is labeled the performance measures. The PM manager needs to
‘‘supply chain strategic planning and performance authorize changes in the definition, to make sure
manager’’ which we here call the ‘‘PM manager’’ that metrics remain consistent between various
for short. areas and with the global scorecard. The metrics
dictionary––although the template is not used ex-
4.4.2. Scorecard format plicitly––remains to be an important element for
The format of the scorecards has changed sig- achieving consistency. ‘‘I do not impose any re-
nificantly since the design project ended. This strictions regarding the format of the functional
confirms that the format is a fluid element for the scorecards, but close coordination regarding defi-
development and implementation of PMSs, nitions is needed’’ says the PM manager. The
while––as we will discuss in Section 4.4.3––the approach of starting with a detailed metrics dic-
individual metrics provide a much more stable tionary was also followed successfully in a more
foundation of the system. The design included recent initiative to further develop the customer
using six clusters and three hierarchical levels. It service functional scorecard. The experiences
took several months after the completion of the confirm the importance of stable metrics that are
design project to actually start the regular pro- coordinated at a detailed level.
duction of a scorecard. Using the originally de-
fined metrics, the format of the report did not 4.4.4. The role of information systems
follow the clustering or the hierarchical layering of Further developments of the PMS will be based
metrics. Actual target and the deviation were re- on the implementation of new information sys-
ported for each measure, and a 3-month average tems. The company will go live with new infor-
was added. Later on a graphical representation mation systems such as SAP, Siebel, and i2 by the
and a color code to signal whether the result was end of 2002, whereas in the current situation data
favorable or unfavorable compared to the target, are gathered in spreadsheets from a wide variety of
were added too. These lengthy reports were used in information systems to create the scorecards.
the functional one-on-one meetings. Experience These scorecards have a fixed format for the users,
showed that it was difficult to use these reports to and information about analysis, actions and re-
focus on exceptions and subsequently taking ac- sults is dispersed across documents and e-mails.
282 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

The new systems should enable managers in dif- • the innovation processes that enable to firm to
ferent countries, business units, functional areas, remain successful in executing processes, deliv-
and at various managerial levels, to create reports ering on service objectives, resulting in sound
from the same data available in a common data- financial performance.
base. The company is testing software for report-
ing. This could lead to a more interactive use of Exploring such linkages and turning understand-
PMSs, where managers aggregate data, drilldown, ing into concrete PMSs that develop over time, is
make new cuts of the data, and see the information the most important theme in the literature. De-
on measurements, analysis and action much more velopment processes are discussed that focus on
clearly together. However, the implementation organizing discussions with different stakeholders
also creates a need for further coordination efforts to obtain their input, and on maintaining and
to make sure that the data required for the various updating the system. Conceptual issues involved in
reports are consistently available. At present, the setting up a PMS are also raised in the literature,
company is in the process of describing perfor- such as the clustering of performance metrics, the
mance metrics and data requirements, and decid- hierarchical structure of a PMS, or the implica-
ing on standards for performance metrics where tions of a PMS for managerial control. Many
necessary. conceptual questions are still not answered, for
The pending implementation of the new infor- example: What are the effects of clustering on de-
mation systems explains why the company is not cision-making processes [28]? What are the effects
investing many resources to further develop the of combining several measures into an overall
current PMS with the Excel spreadsheets. The score? Why do some firms rank, for example,
objective is to get experience with the current way quality improvement opportunities using some
of working, to make sure that these measurements quality measures denominated in financial terms
are possible with the new information systems, and (e.g. euros for rework, scrap, and warranty ex-
to start improving on aspects such as presentation penses), others in percentages or counts (e.g. defect
systems, clustering, aggregation, drilldowns and rates, or customer complaints), and some in survey
sorting the data. scale points (e.g. customer satisfaction indices)
[20]? What are suitable criteria for forming a
4.5. Method cluster (for example, a high correlation between
measures, or usersÕ perceptions of which measures
After having described the design process and belong to the same category)? Is it more important
the experiences during the first year, it is time to to have a broad variety in measures, or is it better
reflect on differences between the way the results to select a smaller set of measures that already
were obtained in the case study and those obtained have a high reliability and predictive validity [20]?
by other approaches. The literature discussed in Furthermore, while the literature discussed in
Section 2 emphasized the development of PMSs as Section 2 often approaches the development and
a design effort to translate strategy into actions, use of a PMS in an ‘‘neutral’’ way for improving
usually at the corporate level. The organizationÕs strategy implementation and operational pro-
financial objectives towards shareholders as well as cesses, it is not clear how the development and use
the strategic actions that have been formulated to of a PMS is shaped by the monitoring and control
achieve those objectives are made concrete in the side of it. ‘‘Political’’ issues such as the reward
measurements and the targets set for each mea- structure tied to the PMS, information asymmetry
sure. These actions deal with three aspects: between management levels and organizational
functions, slack building when setting targets be-
• the service and value the firm seeks to offer its come relevant to consider [34].
customers, Without diminishing the importance of the
• the internal processes that the firm needs to exe- conceptual issues mentioned above, and acknowl-
cute and improve, edging that we did not address these in the present
 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 283

study, a main finding of our study is that other PMS. Working in parallel on four scorecards (for
elements proved to be also important, elements Operations, Transportation, Warehousing, and
that have not received much attention before. In Customer Service) in combination with periodical
our company––and this may actually apply to meetings, created a consistent framework for PM.
many more settings of developing and imple- This eventually resulted in a metric dictionary
menting PMSs in Operations––a key element in listing all metrics (including relevant attributes)
the process was the coordination effort involved in displayed by the four scorecards. Such a document
bringing PM to a higher level. There existed many is important for further development of an inte-
reports, at various organizational levels, both grated PMS.
inside and outside the Operations function. 2. Use a standard metric definition template
Also initiatives to improve PM were going on that includes all relevant metric attributes needed
throughout the organization, and the company to produce or reproduce metric values in a con-
was implementing new information systems and sistent way (see Table 3). Compile a metric dic-
needed to ensure the future provision of the right tionary from the current metric definitions to serve
data. All these circumstances placed coordination as a basis for development and as a reference for
in the center. communication with all parties involved. Con-
Building and sharing a detailed understanding fronting the metrics used with the organizational
of the definitions of performance metrics became objectives, we found gaps in the selection and
crucial to the development and implementation of identified superfluous information.
the PMS. While the literature and our initial ap- 3. Use a clustering that creates a basis for the
proach placed a lot of emphasis on the structure development of performance metrics and supports
and presentation of scorecards (such as clustering communication. The number of clusters as well as
in several perspectives) we found that such dis- the cluster criteria may vary from situation to
cussions––although useful for stimulating the situation.
thinking about the system––were not paramount 4. Feedback on the PMS is more useful if real
for moving forward. During the design effort the data is used. In the case of dummy data, users are
metrics dictionary grew to become the main less motivated to explore the possibilities of the
output and afterwards this has been used to ac- system and its shortcomings.
tually implement the ‘‘Operations Flash’’ without 5. Commitment also means that data specialists,
using the ideas about clusters, aggregation, and ICT staff and other employees able to deliver the
hierarchical structure. Again, we do not want to most reliable data about a process (including in-
suggest that structure and presentation are un- terpretation) are allowed to spend time on such an
important, but that the role of a performance initiative.
metrics dictionary has been underestimated in the 6. Postpone the selection of dedicated PM
literature. software until the basis of the PMS (the metric
dictionary) is mature. This avoids the purchase of
4.6. Some ‘‘lessons learned’’ expensive IT systems that might not bring the ex-
pected improvement in PM.
To conclude this section, we suggest some 7. The adoption and further development of
‘‘lessons learned’’, regarding the development and the PMS requires a PM manager who is a ac-
introduction of a PMS in Operations. We attempt cepted member of the management team that re-
to avoid too obvious suggestions, for example sponsible for the supply chain. The PM manager
about the need to involve all players throughout is responsible for the whole reporting and im-
the process, and we try to reflect on what we provement process, not just as an analyst or ac-
considered surprising compared to our starting countant, but as a manager in charge of concrete
point based on the literature on developing a PMS. follow-ups and monitoring the effects of actions,
1. A cross-functional alignment forum of man- as well as being responsible for improving the
agers and users delivers a basis for an integrated PMS itself.
284 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

5. Conclusions name, objective, scope, target, definition, unit of

measure, frequency, data source, owner, drivers,
In this paper we have described the process and comments. The development of a metrics
and the results of an initiative to significantly im- dictionary may sometimes constitute the main re-
prove the supply chain PM capabilities of a com- sult of a PM initiative, while existing design ap-
pany. The results may be seen as illustrations that proaches place a heavy emphasis on developing
provide guidance for similar undertakings. The reports and scorecards.
clustering, the hierarchical levels, the graphical 3. Our findings suggest that the notion of ‘‘pe-
formats, the template for the data dictionary and riodic review’’ as discussed in the literature could
the practical experiences constitute a relevant ex- be further refined. The review of the metrics
pertise that is new in the literature on PM and themselves is a difficult effort. In our study it was
Operations Management. The paper also contrib- considered worthwhile to invest significant re-
utes to the theoretical knowledge of PM, as we sources in developing a standardized, shared set of
have used our empirical findings to reflect on the performance metrics to be used across the supply
literature. This leads to four conclusions: chain. On the other hand, reviewing and improv-
1. The literature describes the development of a ing the reports and scorecards could be done on a
PMS as a design effort to translate strategy into ac- continuous basis. The selection of individual met-
tions. There is much attention for exploring linkages rics, the way in which metrics are clustered, the
between various dimensions of performance and hierarchical ordering of clusters, and the way of
turning that understanding into concrete PMSs. presenting the reports are all aspects that are easily
Processes that are discussed focus on obtaining in- changeable from one month to the other (or other
put from different stakeholders and maintaining and reporting intervals). We suggest that every time
updating the system. There is not much explicit at- when a supply chain management team is review-
tention for earlier and parallel PM initiatives. Such a ing a performance report, these users may also
‘‘green field’’ approach did not work particularly provide feedback on the report itself, directly to
well in our case study, and we believe that this has the person responsible for that report. In this way
more general implications. Existing reports at vari- the development of the PMS becomes a continu-
ous levels, both inside and outside the operations ous effort, based upon a far less frequently chan-
function, place constraints on current PM. They ged basis of standardized metrics.
provide opportunities at the same time, because 4. Several techniques can be used to reduce
measurements developed elsewhere can be incor- complexity of a PMS, such as clustering of metrics
porated. Our findings suggest that developing PMSs into various perspectives, hierarchical structuring
should to a large extent be understood as a coordi- of reports, and aggregating various performance
nation effort to understand current metrics in detail, measures into one number. This aggregation can
to identify shortcomings, and to include ongoing be done directly if the underlying metrics are ex-
initiatives that affect PM (such as new information pressed in the same units of measure, such as
systems, parallel initiatives for developing PMSs, monetary units, units of products, etc. However, in
and global scorecard development). Much of the our study each cluster contained a number of in-
existing literature focuses on design efforts at the dividual metrics with different dimensions. Yet
corporate level, while our study is at the operations there was a need to report the overall performance
or supply chain level. This may explain why coor- of a cluster in one number. So we used an intuitive,
dination is such an important factor in this study. easy to use method for normalization of metrics. It
2. The need for co-ordination creates a central derives from [31]. Linear normalization has been
role for a set of shared and clearly defined per- chosen for its simplicity. However, experiences in
formance metrics. This role has not received much this study showed that such techniques are not
attention in the literature so far. Relevant attri- crucial for getting started with a PMS. This rein-
butes for describing the metrics and building a forces the conclusion that coordination through
‘‘metrics dictionary’’ identified in this study were: metrics definition is essential early in the process,
 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 285

while improving the structure and presentation References

scorecards is a more continuous process.
[1] M. Abrahamsson, S. Brege, Structural changes in the
supply chain, International Journal of Logistics Manage-
Clearly, there are limitations to the research
ment 8 (1) (1997) 35–45.
approach followed in this study. As with any case [2] E. Anderson, G.S. Day, V.K. Rangan, Strategic channel
study, the findings cannot easily be generalized to design, Sloan Management Review 38 (3) (1997) 59–69.
other empirical settings. The approach initially [3] P. Andersson, H. Aronsson, N.G. Storhagen, Measuring
taken may have worked well in another context, logistics performance, Engineering Costs and Production
Economics 17 (1989) 253–262.
while the approach that was subsequently devel-
[4] B.M. Beamon, Measuring supply chain performance,
oped here may not be transferable to another International Journal of Operations & Production Man-
company. That is why we have included a de- agement 19 (3) (1999) 275–292.
scription of the considerations that led us to our [5] B.M. Beamon, T.M. Ware, A process quality model for the
particular development process. It opens the pos- analysis, improvement and control of supply chain sys-
tems, Logistics Information Management 11 (2) (1998)
sibility to assess the applicability of our approach
105–113.
in another setting where PM needs to be improved [6] C. Berliner, J.A. Brimson, Cost Management for TodayÕs
for operations across a supply chain. Also, the Advanced Manufacturing, Harvard Business School Press,
outputs produced in this project and the practical Cambridge, MA, 1988.
experiences gained have been listed. They may [7] M. Bourne, J. Mills, M. Wilcox, A. Neely, K. Platts,
Designing, implementing and updating performance mea-
facilitate reflection upon the process and indicate
surement systems, International Journal of Operations &
how certain things may be done differently in an- Production Management 20 (7) (2000) 754–771.
other project. However, we did not perform such [8] D.J. Bowersox, D.J. Closs, Logistical Management. The
tests. Having said this, we believe that our findings Integrated Supply Chain Process, McGraw-Hill, New
have relevance for companies who are imple- York, 1996.
[9] A.M. ten Broeke, Prestatie––indicatoren in de logistiek:
menting a PMS and already have existing PMSs.
Aanpak en samenhang, Kluwer/NEVEM, 1998 (in Dutch).
Operations Management across supply chains is [10] R.D. Buzzel, G. Ortmeyer, Channel partnerships stream-
a topic that rightly receives much interest in practice line distribution, Sloan Management Review 36 (2) (1995)
and in the operations literature. It is clear that PM is 85–96.
an important element of Operations Management. [11] G. Chow, T.D. Heaver, L.E. Henriksson, Logistics per-
formance: Definition and measurement, International
This study provides empirical findings on the de-
Journal of Physical Distribution & Logistics Management
velopment of PMSs in Operations Management of 24 (1) (1994) 17–28.
supply chains. Despite the limitations mentioned [12] G.B. Davies, Strategies for information requirements
above, we believe that the contribution of the paper determination, IBM Systems Journal 21 (1) (1982) 4–30.
to the literature on PM and Operations Manage- [13] R.G. Eccles, The performance measurement manifesto,
Harvard Business Review 69 (January–February) (1991)
ment is relevant. With the insight gained in this case
131–137.
study, we suggest several questions for further re- [14] S.E. Fawcett, M.B. Cooper, Logistics performance mea-
search: Do the users want to work with normalized surement and customer success, Industrial Marketing
scores or do they want to see actual numbers? Do Management 27 (1998) 341–357.
users prefer a printed version instead of the inter- [15] L. Fitzgerald, P. Moon, Performance measurement in
service industries: Making it work, Chartered Institute of
active one? How can the creation of scorecards be
Management Accountants, London, 1996.
automated? What is the best way to link the system [16] S.D. Flapper, L. Fortuin, P.P.M. Stoop, Towards consis-
to other sources of information? Does the system tent performance measurement systems, International
fully satisfy the needs of the users? Journal of Operations & Production Management 16 (7)
(1996) 27–37.
[17] L. Fortuin, Performance indicators––Why where and
how? European Journal of Operational Research 34
Acknowledgements (1988) 1–9.
[18] J.C. Fransoo, M.J.F. Wouters, Measuring the bullwhip
The authors like to thank Nike for their co- effect in a supply chain, Supply Chain Management 5 (2)
operation in publishing the results of this study. (2000) 78–89.
286 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286

[19] R.W. Hall, H.T. Johnson, P.B.B. Turney, Measuring up: [31] B.H. Maskell, Performance measurement for world class
Charting pathways to manufacturing excellence, Business manufacturing. A model for American companies, Pro-
One Irwin, Homewood, IL, 1990. ductivity Press, Cambridge, MA, 1991.
[20] C.D. Ittner, D. Larcker, Innovations in performance [32] D.M. McCutcheon, J.R. Meredith, Conducting case study
measurement: Trends and research implications, Journal research in operations management, Journal Of Operations
of Management Accounting Research 10 (1998) 205– Management 11 (3) (1993) 239–256.
238. [33] D. Medori, D. Steeple, A framework for auditing and
[21] H.T. Johnson, R.S. Kaplan, Relevance lost: The rise and enhancing performance measurement systems, Interna-
fall of management accounting, Harvard Business School tional Journal of Operations & Production Management
Press, Boston, MA, 1987. 20 (5) (2000) 520–533.
[22] R.S. Kaplan (Ed.), Measures for manufacturing excellence, [34] K. Merchant, Modern Management Control Systems,
Harvard Business School Press, Boston, MA, 1990. Prentice Hall, 1998.
[23] R.S. Kaplan, D.P. Norton, The balanced scorecard–– [35] A.J. Nanni, J.R. Dixon, T.E. Vollmann, Integrated
measures that drive performance, Harvard Business Re- performance measurement: Management accounting to
view 70 (January–February) (1992) 71–79. support the new manufacturing realities, Journal of Man-
[24] R.S. Kaplan, D.P. Norton, Putting the balanced scorecard agement Accounting Research 4 (1992) 1–19.
to work, Harvard Business Review 71 (September–Octo- [36] A.D. Neely, M. Gregory, K. Platts, Performance measure-
ber) (1993) 134–147. ment system design, International Journal of Operations &
[25] R.S. Kaplan, D.P. Norton, Using the balanced scorecard Production Management 15 (4) (1995) 80–116.
as a strategic management system, Harvard Business [37] A.D. Neely, A.H. Richards, J.F. Mills, K.W. Platts,
Review 74 (1) (1996) 75–87. M.C.S. Bourne, Designing performance measures: A
[26] R.S. Kaplan, D.P. Norton, Having trouble with your structured approach, International Journal of Operations
strategy? then map it, Harvard Business Review 78 (5) & Production Management 17 (11) (1997) 1131–1152.
(2000) 167–178. [38] N.G. Olve, J. Roy, M. Wetter, Performance drivers: A
[27] J. Lampel, H. Mintzberg, Customizing customization, practical guide to using the balanced scorecard, John Wiley
Sloan Management Review 37 (4) (1996) 21–29. & Sons, New York, 1999.
[28] M. Lipe, S. Salterio, A note on the judgmental effects of [39] R.J. Ormerod, Combining management consultancy and
the balanced scorecardÕs information organization, research, Omega, International Journal of Management
Accounting, Organizations and Society 27 (2002) 531– Science 24 (1) (1996) 1–12.
540. [40] M. Sanger, Supporting the balanced scorecard, Work
[29] F.A.B. Lohman, The effectiveness of management infor- Study 47 (6) (1998) 197–200.
mation, Ph.D. Thesis, Delft University, 1999. [41] G.D. Scudder, C.A. Hill, A review and classification of
[30] R.L. Lynch, K.F. Cross, Measure up!: Yardsticks for empirical research in operations management, Journal of
continuous improvement, Blackwell, Oxford, 1995. Operations Management 16 (1998) 91–101.