Professional Documents
Culture Documents
Designing A Performance Measurement System: A Case Study: Clemens Lohman, Leonard Fortuin, Marc Wouters
Designing A Performance Measurement System: A Case Study: Clemens Lohman, Leonard Fortuin, Marc Wouters
www.elsevier.com/locate/dsw
Case Study
a
TLO Holland Controls bv, Noordhoek 37, 3351 LD Papendrecht, The Netherlands
b
Faculty for Technology Management, Eindhoven University of Technology, Paviljoen F.18, P.O. Box 513,
NL 5600 MB Eindhoven, The Netherlands
c
School of Technology and Management, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands
Received 11 January 2002; accepted 26 November 2002
Abstract
Performance measurement (PM) by means of local performance indicators (PIs) is developing into performance
management at a company-wide scale. But how should PIs at various levels in the organization be incorporated into
one system that can help managers, working at levels that range from operational to strategic? How do we convince
potential users and obtain their support when starting to develop such a system? How can we aggregate PIs? How do we
present results? This paper addresses these and related questions. It is based on a case study carried out at the European
Operations department of Nike, a company producing and selling sportswear worldwide. The study resulted in a
prototype system that basically is a balanced scorecard tailored to the needs of the company. The empirical findings
differ in some ways from the literature on developing performance measurement systems (PMSs) in Operations. Dis-
cussing these differences provides new theoretical and practical insights. They relate to the role of parallel initiatives for
PM, the role of standardized metrics, the continuous improvement of PMSs, and the normalization and aggregation of
measures. Our findings suggest that developing PMSs should to a large extent be understood as a co-ordination effort
rather than a design effort. The lessons learned cannot have universal validity, but may be helpful in similar kinds of
initiatives.
2003 Elsevier B.V. All rights reserved.
0377-2217/$ - see front matter 2003 Elsevier B.V. All rights reserved.
doi:10.1016/S0377-2217(02)00918-9
268 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286
Several developments have created a need for The literature on PM in operations describes
companies to improve their supply chain man- several methods for developing PMSs. A charac-
agement. First, cross-functional co-operation teristic of many of these methods is the focus on
needs to be improved along the supply chain to developing performance metrics and a PMS based
offer shorter delivery times, more flexibility and on the firmÕs strategy and processes (see for ex-
faster introduction of new products (see, e.g. ample [30]). The literature also addresses the
[2,10,27]). Many companies are organized func- comparison of desired performance measures with
tionally, i.e. around subsequent stages of produc- existing measures (to identify which current mea-
tion, which makes it difficult to control the supply sures are kept, which existing measures are no
chain. Serving customers better requires synchro- longer relevant, and which ‘‘gaps’’ exists so new
nization of functions such as marketing, sales, measures are needed, [33]) and the periodic revi-
distribution, manufacturing, and purchasing. Sec- sion of PMSs once implemented [7]. However, the
ond, better synchronization is not only important literature does not provide a good understanding
across functional boundaries, but also across na- of how the process of developing a PMS is im-
tional boundaries. Spanning these boundaries has pacted by existing PMSs, both within and outside
especially occurred in Europe, where many com- the operations, at a more fundamental level. The
panies have moved from strong national organi- objective of this paper is to provide empirical re-
zations with local production, products and sults on improving PMSs to support supply chain
customers, to an organization where production management, using a case study methodology. A
has become more specialized and one factory comparison of these empirical results with the lit-
serves a specific part of the product range for the erature provides new theoretical insights. The
whole of Europe. Sales and marketing have be- findings are based on a case study within the Eu-
come partly centralized. This moves demand ropean Operations function of Nike. Case study
management, product allocation, marketing, and research has a small but consistent tradition in the
distribution to a European level. So there is a need operations management literature. Scudder and
to manage the supply chain on a European scale Hill [41] reviewed empirical research in operations
[1]. A third development is that streamlining of management published in the years 1986–1995 and
Operations across a chain of separate companies concluded that the amount of published empirical
has become more important because this creates work increased, both in absolute number and as a
opportunities to offer better service to end- percentage of the total number of articles pub-
consumers against lower costs for the supply chain lished. They found that case study and survey are
in its totality. However, information systems for the most widely used research designs, with a
costing and PM have generally not been very consistent mix of both designs and surveys being
helpful for managing operations, because such used more extensively compared to case studies.
systems were based on overly simplified models of The theoretical contribution of this paper is to
manufacturing activities and resource consump- show the limitations of a ‘‘green field’’ approach in
tion, which produced inaccurate cost data. More- the development of PMSs. The presence of existing
over, in many companies there was a lack of measures and parallel PM initiatives may quite
non-financial measures [21]. fundamentally change the development from a
Four terms will be used throughout the paper. ‘‘design approach’’ to a ‘‘coordination approach’’
A performance indicator (PI) is a variable that focused at aligning the supply chain operations
expresses quantitatively the effectiveness or effi- PMS with existing performance measures and
ciency or both, of a part of or a whole process, or parallel initiatives outside the operations function.
system, against a given norm or target [17]. PM is Our findings point to the central role of a shared
the activity of measuring performance using PIs. A set of standardized performance metrics as a tool
PMS is a system (software, databases, and proce- for achieving such coordination.
dures) to execute PM in a consistent and complete The paper has the following structure. In Sec-
way. A PI also is called ‘‘performance metric’’. tion 2 we look at the literature relevant to our
C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 269
about strategy, customers, processes, etc. becomes typical steps such as defining a firmÕs manufac-
available. The appropriate measures are derived turing strategy through competitive priorities,
from such information in several rounds to review linking success factors to the competitive priori-
and revise the measures. The availability of data is ties, defining measures, implementation of mea-
one of the considerations in the design process. sures, and executing periodic maintenance. The
There is also much attention for updating perfor- ‘‘audit’’ step may lead to eliminating some existing
mance measures once they have been imple- measures and identifying gaps in the current
mented. Kaplan and Norton [24] emphasize using measurement system. In this paper, however, we
documents, interviews, and executive workshops describe an empirical study that shows how the
for gathering information and building consensus. presence of existing measures and parallel PM
Various approaches can be used to design the PMS initiatives may have more fundamental impact on
([29] based on [12]): the development of PMSs. Rather than creating
1. Asking: Techniques to find out the require- the need for an additional step to verify and ad-
ments of managers, such as interviews, group dis- just, it may change the nature of the development
cussions, planning meetings, and surveys, are most challenge from (a) designing a PM ‘‘as if’’ from a
often used to develop a PMS. clean sheet of paper to (b) making sure that an
2. Prototyping: Instead of focusing primarily improved Operations PMS is aligned with existing
on a thorough analysis of the information needed, performance measures and parallel initiatives
an initial set of requirements is specified and a outside the operations function. So far, this has
prototype system is built. Through interaction received little attention in the literature on PM in
with users of the system (managers), require- Operations.
ments are added or changed until the user is sat- In summary, PM yields a fundamental type of
isfied. management information needed for controlling
3. Planning methods: Methods that design ap- operations. It creates focus, triggers corrective
propriate measures based on the characteristics of action, is the basis for evaluating performance,
the firm, such as strategy, processes, and custom- and may help challenging and improving strategic
ers. For example, a method could be followed to choices. Both the management accounting litera-
determine a few areas (critical success factors) that ture and the operations literature focus on the
dictate the success of the firm. For such areas connections between strategy and PM––the role of
critical success factors are described, which leads PM as translating strategy into concrete goals and
to the definition of measures that capture these monitoring the delivery of strategy––and between
factors. Operations and PM––measures need to capture
4. Existing reports: Often a useful source of the relevant characteristics of the underlying op-
information to be used to design the PMS. erational processes. Approaches for developing
Implicit in many approaches for designing PMSs use various ways to gather information, and
performance measures is a ‘‘green field approach’’ there is much attention for an iterative process in
that does not pay explicit consideration to existing which measures are developed and adjusted as
measures. However, in many settings it is realistic more information becomes available about strat-
to acknowledge that reports relevant for managing egy, customers, processes, and the availability of
operations already exist at various levels in the data. There is also much attention for updating
organization, within and outside the operations performance measures once they have been im-
function. Medori and Steeple [33] is one of the few plemented. However, there is far less literature that
papers that explicitly discuss the relationship with provides an understanding of how the process of
existing measurement systems; their design method developing a PMS is impacted by existing mea-
includes an ‘‘audit’’ step to compare the newly sures (or new measures that are being developed
defined desired performance measures with the simultaneously as a result of other initiatives) at
measures that already exist. However, this is various levels both within and outside the opera-
merely one element of a process that includes tions function.
272 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286
3. Supply chain performance measurement cohesion. This makes it hard to focus attention in
an effective way and causes a lack of overview.
Many companies are trying to improve supply This not only makes management to feel discom-
chain PM, in order to support managing opera- fort, but it also can lead to missing opportunities.
tions across supply chains. It may be useful to 3. Uncertainty what to measure––Often uncer-
think first of two extremes: (1) several functional tainty exists about what exactly should be mea-
or regional departments are each responsible for sured on a supply chain level. Since current reports
one aspect or one part of the supply chain and mainly cover parts of the supply chain, it is likely
only top management is responsible for the total that certain high-level metrics are lacking. This
financial results; and (2) a situation in which a adds to the managerÕs discomfort.
management team is responsible for the overall 4. Poor communication between reporters and
performance of the whole supply chain. Most users––Communication between the creators and
companies are somewhere in between. But as users of reports is often poor. The creators often
companies move towards a more integrated Op- hardly know their audience and the exact purpose
erations Management function across the supply of the reports. This results in poor readability and
chain, it becomes necessary to measure the per- limited usefulness. The users on the other hand
formance of the various parts of the supply chain sometimes do not know why they receive a certain
on various dimensions, in a consistent way. There report and so they do not use it at all. The lack of
is a need to define and measure performance for interaction make the reports outdated in relation
the supply chain as a whole and to be able to to the business as well as user preferences.
drilldown to different measures and different levels 5. Dispersed IT infrastructure––Companies use
of detail, in order to understand the causes of many information systems that are linked in some
significant deviations of actual performance from way. The dispersed IT infrastructure produces a
planned performance. However, many companies number of issues. Firstly, it adds to the lack of
seem to be facing serious difficulties in imple- data integrity between the reports. Since consid-
menting such supply chain-wide PMSs that cap- erable overlap exists between the systems, certain
ture various dimensions of performance at various data can be extracted from multiple sources and
levels in a consistent way. These difficulties have this often leads to inconsistency. Secondly, the
various causes: infrastructure does not provide visibility over the
1. Decentralized, operational reporting history–– supply chain, owing to the absence of connectivity.
There is often a history of decentralized reporting Thirdly, certain systems are not designed for re-
with a focus on local operational use within facto- porting uses or cannot provide data at reasonable
ries, transportation linkages, distribution centers, cost at all.
sales offices, etc. This has led to an uncontrolled These five complexes of difficulties raise the
growth of reports with many inconsistencies. These question how supply chain PM can be improved.
inconsistencies have to do with definitions of per- The objective of this paper is to report on an
formance metrics, sources of data for obtaining empirical study on the development of PM. By
measures, and ways of presenting reports. Man- doing so reflection on current theories becomes
agers who try to construct a total picture of the sensible.
supply chain from these reports find themselves
confronted with a large volume of (inconsistent)
information in a format that does not support 4. The case study
integrated analysis.
2. Deficient insight in cohesion between metrics–– The study took place at European Operations
Since current reporting has an operational focus, of Nike. The company was continuously improv-
the metrics are used to monitor sub-processes of ing its supply chain management. As part of these
the supply chain. These pieces of information are efforts management decided to assign some of
analyzed on an individual basis rather than in their resources for improvement projects to PM.
C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 273
Especially the integration of various local PI into a rope in 1977. Nowadays, there is one European
company-wide consistent system for PM was re- distribution center, the Customer Service Center
quired. The authors were involved through a de- (CSC) in Laakdal, Belgium. In 1999, Customer
sign project carried out as part of a university Service––responsible for order and query man-
program in international logistics. One author agement––also was concentrated in one place, the
carried out the project as a postgraduate student European Headquarters (EHQ) in Hilversum, The
to complete this program; the other two authors Netherlands.
coached and supervised the work. The project was The company is organized around three lines of
done at the company full-time during six months business: footwear, apparel, and equipment.
and can be described as action research [39]. The Footwear and apparel make up the largest part of
empirical observations and experiences are com- the business; they are almost equal in size.
pared to previous theory regarding the develop- Equipment is relatively small (5% of revenue),
ment of PMSs in order to develop a better but it is growing fast. The lines of business are
understanding of how such theories or concepts divided into product lines, and each line is divided
might work in real situations. This might be con- into categories. The total assortment per line of
sidered as a descriptive case study, or maybe an business can be characterized as ‘‘large’’. This
exploratory case study [32]. A group of about 10 holds in particular for apparel: it comprises of
company managers reviewed the results every few 60,000 stock keeping units in the supply chain.
weeks, and in-between such review meetings, there Comparable figures for footwear and equipment
were interviews with many more company man- are 25,000 and 1000, respectively. The product life
agers and frequent informal discussions with the cycles are short, which is normal in the clothing
company coaches and the director of Operations. industry. Most products are specially designed for
This paper reports on the situation at the end of a specific season. This holds less for equipment
the project. The company has continued the de- and not for basic products like socks, white shirts,
velopment and implementation of the PMS to etc.
modify the clustering of metrics, to increase data The European region comprises Europe, the
availability so that more metrics can actually be Middle East, and Africa (EMEA). The business
measured and reported reliably, and to change the volume in the last two areas is very small as
structure and presentation of the scorecards. compared to the European volume. They can,
First we will introduce the company in Section however, be considered as emerging, high potential
4.1. The design of the PMS is described in Section areas. The ‘‘big five’’ in Europe (UK, France,
4.2, while the development approach and the Germany, Spain, and Italy) make up 70% of the
critical choices made in this process are discussed total revenue of Nike Europe.
in Section 4.3. Next we discuss the companyÕs ex- Uncertainty of demand is an important char-
periences during the first year of implementing and acteristic. Although market intelligence is widely
using the PMS, in Section 4.4. In Section 4.5 we present and aggressive marketing is being used,
discuss the method and how our empirical findings consumer behavior is hard to predict when it
suggest that elements previously not pointed out in comes to fashion. Although only a small portion
the literature may be critical for the development of products is delivered directly to consumers, via
and implementation of a PMS in Operations. To the Internet (www.Nike.com) and Nike retail, their
conclude, Section 4.6 lists some practical sugges- buying volume does affect the sales of Nike to
tions derived from this project. retail organizations.
This paper does not deal with the Nike supply
4.1. The company chain in its totality: it covers the demand for
the European market (sourced worldwide) and
Nike is active in the clothing and sport acces- is restricted to Operations, which consists of
sory industry. After becoming successful in the Transportation, Warehousing, and Customer Ser-
USA, Nike started exporting its products to Eu- vice. By putting together various requests from
274 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286
management, the project objectives became as the metric selection. It resembles the balanced
follows: scorecard, but it is extended with a cluster for
Sustainability and one for People. This extension
1. to develop a set of high-level performance met- is made in order to fit OperationsÕ specific char-
rics tailored to the specific business needs for acteristics and to pay explicit attention to these
use by the senior supply chain management areas. Fig. 2 depicts the clusters together with the
team, i.e. Operations, while including existing questions that should be answered by the metrics
local metrics as much as possible and sensible; included. Clearly, mission and strategy are the
2. to design a format, i.e. a scorecard, displaying starting point and source for objectives in the six
the metric scores at the level of Nike as well clusters. All relevant areas for Nike Operations are
as that of the business units. represented. Here are some details:
Customer––Nike Operations is connected to its
4.2. The design customers by means of a physical process (the
delivery of products) and an informational process
Our case study resulted in a new prototype (via CSRÕs, i.e. Customer Service Representatives).
system for PM. Its scope is limited to Nike Europe By using information directly obtained from the
and the Operations function, including Ware- customers as well as information about the pro-
housing, Transportation and Customer Service. cesses on the interface between Nike and its cus-
Point of departure is a set of design guidelines that tomers, the performance towards the customers is
are tailored to NikeÕs characteristics. Application measured on aspects such as customer satisfaction,
of these guidelines produced a set of performance shipment queries, and order fill rate.
metrics, and a scorecard for displaying the corres- Sustainability––This cluster contains metrics
ponding information. that relate to the interaction between Nike and its
environment. In the recent past, the company has
4.2.1. Performance metrics started several projects to increase the awareness
The metric selection should contain output- for sustainable growth.
related PIs as well as (leading) operational indi- Financial––The Financial cluster offers a view
cators. Following this guideline of the design on OperationsÕ contribution to shareholder value
approach, we developed a clustering method for by looking at costs and revenue and margin in-
fluencing factors. Since Operations is a cost center, button on the bottom allows updating of the
it can contribute to shareholder value by realizing gauges according to changes in the data. If a user
low costs and facilitating the generation of reve- wants more information about the performance of
nues. The financial aspect includes cost per unit a cluster, he can click on the button below the
measures and inventory-related measures. Abso- corresponding gauge to enter a lower information
lute costs are not included since other mechanisms layer.
are present to monitor cost versus budget. Mid level––The next level in the scorecard
Process Improvement––The Process Improve- shows the highest-level indicators for the selected
ment cluster contains metrics that relate to long- cluster. Fig. 3B shows this for the cluster called
term improvement trajectories and strategic issues, Customer. The overall cluster score is repeated in
such as the progress of key projects, the quality of the black box. The score of each key PI is depicted
supply, and the complexity of Operations. The numerically and graphically by the bar chart. The
score of this cluster needs to be sufficient in order upper bar indicates the score of the current month;
to safeguard growth in the future. the lower bar that of the previous month. The user
Product Flow––The Product Flow cluster con- can find comments on the scores by using the pull-
tains metrics that track effectiveness in the supply down list underneath the black box. Once again,
chain. Basically, the performance of this cluster definitions can be found by clicking the button in
forms the basis of the performance of the delivery- the right corner on the bottom. The ‘‘back’’ button
related metrics in the Customer cluster. Accuracy in the upper right corner returns the user to the
and throughput of good flows at the subsequent dashboard. Clicking the buttons on the right of the
stages of the supply chain are the focus in this bar chart saying ‘‘details’’ takes the user a level
cluster. down in the hierarchy. Depending on the position
People––The organizational health can be as- in the hierarchy another mid-level screen appears
sessed by means of this cluster, such as employee or a lowest-level screen is reached.
satisfaction, professional development, and diver- Lowest level––This level provides the user with a
sity. presentation of the performance of an indicator
that is tailored to its characteristics. An important
4.2.2. Scorecards common characteristic for the graphs on this level
A structure with three layers is used for dis- is that they show the development over time. In
playing the information. The characteristics per Fig. 3C we see the development of ‘‘CPU Total’’
layer are described below. The scorecard prototype on the scorecard issued in the beginning of Jan-
is built in Microsoft Excel with supplementary uary 2001 covering performance up to December
programming in Visual Basic. 2000. The monthly actual values are displayed
Top level––If a user opens the scorecard file, the together with a 12-month moving average. The
screen will show the highest level of the PI struc- latter cancels out seasonal fluctuations and facili-
ture: the metric clusters (see Fig. 3A). We call it the tates trend analysis. The seasons are depicted
‘‘dashboard’’ as, like in a car, it displays high-level, below the graph as a reference. The figure shows
aggregated performance information. The gauges the CPU for the total business; the user can choose
depict the score of each cluster numerically and to view CPU for individual business units by
graphically. The main pointer indicates the score clicking one of the buttons in the upper right box.
of the current month, a shadow pointer that of the The way of displaying performance used here is
previous month. In this way the user can see how specific for this indicator. For other metrics other
the metric value develops. The color of the cluster graphical formats are used if more appropriate.
names on the buttons underneath the gauges in- Metrics and scorecards are the key elements of
dicates whether the underlying metrics are out of our system. Therefore, they have been discussed in
their control range. The buttons on the left side of some detail. Other aspects, however, are important
the dashboard give access to a user guide and to a as well: normalization and aggregation, usage and
list of metric definitions used in the scorecard. The maintenance.
276
C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286
Fig. 3. Scorecards at three levels. (A) Top level; (B) mid level; (C) lowest level.
C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 277
that the design approach has to be followed • Headquarters was taking initiatives to develop
each year to assure an up-to-date PM tool. a worldwide scorecard for the Operations func-
tions. Zooming-in on Europe should provide
4.3. The development approach information that would be consistent with the
scorecards that European Operations would
In view of the literature on PM, rapid proto- use themselves.
typing seemed a logic start. The idea was to look at
current reports and add some potentially relevant Furthermore, collaboration with the business ana-
measures to produce a first version of a report. lysts who produced current reports appeared more
Future users then would review this, i.e. the time consuming than anticipated. The prototyping
management team that was responsible for the approach works if gradually more of the proposed
supply chain. A prototype could give concrete metrics are presented on the basis of real data and
examples of how the new PMS would look and actual measurements. But many people placed
this could stimulate discussion among the users demands on the business analystsÕ time. Producing
and generate feedback to the designer. Next, by credible numbers instead of merely ‘‘theoretical’’
using the comments of the users, improvements exercises took a lot of time.
would become possible, until eventually a satis- Hence, a new approach was chosen, in close
factory system is obtained that can be made collaboration with management at the European
available to users at various levels and locations in Headquarters. Central to the new approach be-
the organization. The prototyping would also be came the development of a metrics dictionary with
used to experiment with software designed for some 100 metrics. The discussions stepped away
producing reports. However, it soon became ap- from the presentation and structure of scorecards.
parent that there were reasons for taking a differ- It focused on getting a detailed understanding of
ent approach. the individual metrics that were used as part of the
First of all the design efforts had to be aligned various ongoing initiatives described above. At the
with other performance reporting initiatives. Nike start of the project metrics were often documented
had already functional scorecards and periodic in an ambiguous way causing failures in commu-
reports (about 140 in total), also outside Opera- nication between reporting employees and man-
tions. Here are some details: agers. In order to increase the quality of the
metrics and communication about the information
• Many reports existed at lower operational levels they list, a template was developed. We followed
and when possible the existing metrics would be Neely et al. [37] and added some more attributes
used, which required a careful understanding of (see Table 3). In other words: more work was done
these metrics. and more time spent before first results were
• The various functions within Operations (Ware- shown to potential users. This process of using the
housing, Transportation, Customer Service) new template and a metrics dictionary caused vivid
had developed functional scorecards, and these discussions about what had been reported in the
would form the main experience base for de- past and how this should be in the future. Fur-
signing the supply chain-level scorecards. In a thermore, template and dictionary made all re-
way, the supply chain-level scorecard would porting employees more aware of PM, which can
be a combination of the functional scorecards be seen as a positive side effect.
enriched with measures that would be pertinent In this new approach, reporting has become a
at the supply chain-level. process of continuous improvement, but in a dif-
• Outside Operations other functions were devel- ferentiated way. On the one hand, the metrics
oping measures that could be incorporated in dictionary has been fixed. This is the central ele-
the supply chain scorecard, such as measures ment to ensure co-ordination between the various
from the Human Resource function about efforts that are going on and being developed over
training and retention. time. On the other hand, the structure of reporting
C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 279
Table 3
Metric definition template: attributes with their explanation (based on [37] with elements added)
Metric attribute Explanation
Name Use exact names to avoid ambiguity
Objective/purpose The relation of the metric with the organizational objectives must be clear
Scope States the areas of business or parts of the organization that are included
Target Benchmarks must be determined in order to monitor progress
Equation The exact calculation of the metric must be known
Units of measure What is/are the unit(s) used
Frequency The frequency of recording and reporting of the metric
Data source The exact data sources involved in calculating a metric value
Owner The responsible person for collecting data and reporting the metric
Drivers Factors that influence the performance, i.e. organizational units, events, etc.
Comments Outstanding issues regarding the metric
definitions and reporting formats, and to ex- concluded later, the power of this tool for dis-
changing ideas about developing and using PM in playing scorecards in many sophisticated ways
the organization. The results of these sessions were distracted the attention from the underlying met-
documented in the metric dictionary. These dis- rics. It also tended to lead to high expectations and
cussions, thus, became the focal point of building a lot of debate about the displaying format. An-
an understanding of the existing measures and other disadvantage was that the tool would be-
reports, and of parallel initiatives to develop a new come expensive if it had to be made available for
PMS. The new PMS was synchronized with ex- many users of reports. Such a wide introduction of
isting procedures and performance reports mainly new software in the company involved much more
by entering detailed definitions of existing and complicated decision-making processes. Experi-
proposed performance measures in the metric menting with the tool caused delays and took too
dictionary. We expected that such synchronization much time from the researchers. After a couple of
would be complex. It turned out to be not so dif- months it was decided to take a step back and only
ficult, because measures in the existing reports al- use a generally available spreadsheet tool, MS
though presented in different ways, used closely Excel.
related definitions, or it was easy to agree on a
common definition. In Section 4.4 there is more 4.4. Experiences
information on how this process of coordinating
existing and new measures continued after the The company has continued developing and
development projected reported here. implementing the PMS after the initial design
Next to the forum, managers, data specialists project described in this paper. They have experi-
and ICT staff were consulted regularly during all mented with different ways of clustering the met-
stages of the development process. Especially rics and presenting the scorecards. They have also
management, being the ultimate user, appeared to gone further in gathering the data required for
need a sense of ownership in order to use the re- actually measuring and reporting all metrics in a
sulting product. It was interesting to see that, when reliable way. The following observations sup-
using dummy data, managers seemed less moti- port the notion that the development of PMSs in
vated to contribute to the development process Operations is a continuous and experimental pro-
than when using real data. In order to get their full cess.
attention, we decided to discuss only those metrics
for which at least one real figure could be shown. 4.4.1. The role of a PM manager
The data specialists and ICT staff were necessary One person of the cross-functional initiatives
to develop metrics that can be reported at rea- group manages the performance reporting process.
sonable cost. In most cases, the employees able to He collects data from various sources, publishes
deliver the most reliable data (including interpre- the monthly reports, discusses the results with
tation) about a process are directly involved in that managers and manages the agenda of follow-up
process. Apart from the fact that they are indi- actions, and continues to develop the PM method.
rectly asked to deliver information about their The scorecard is reported every month. In the
own performance, often no time is explicitly bud- second or third week of the month the functional
geted for this activity. Co-operation with these scorecards (transportation, warehouse, customer
employees appeared to be cumbersome and man- service) and a so-called ‘‘Operations Flash’’ are
agement needed to communicate the priority of completed. In the following week the ‘‘Operations
the project several times in order to improve this. Flash’’ is discussed in one-on-one review meetings
The last category of decisions concerned the between the managers who are responsible for the
tools used. Early in the project a start was made functional scorecards and the manager of the
with the use of a software tool that helps to cal- performance reporting process, whereby the func-
culate and visualize the performance on a hierar- tional scorecards are used as background infor-
chical set of metrics. However, as could be mation. In the third or fourth week, always on a
C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 281
Monday, the ‘‘Operations Flash’’ is discussed in tions. This led to the development of the Opera-
the Operations Staff meeting. The next day tions Flash report, a one-page report showing the
(Tuesday) the Operations Flash report and the metrics with a focus both on business unit and on
action items that have been taken up in the Op- countries. As expected, the use of the PMS trig-
erations Staff meeting are discussed with world gered a process of continuously revising the format
headquarters in the monthly telephone conference of presenting the scorecards.
with the regional management team (the so-called
regional logistics call). Every quarter a global 4.4.3. Coordination of initiatives
scorecard is discussed more extensively in the The topic of coordination between various PM
quarterly global supply chain leadership meeting. initiatives remains to be a central element of de-
The global Operations scorecard shows a set of veloping and implementing PMSs. This has be-
measures for four different regions (EMEA, US, come apparent for the implementation of a global
Asia Pacific, Canada). The PMS has become a tool Operations scorecard. Some measures that had
that is actually being used in various meetings and been developed in this project (i.e. for the EMEA
one-on-one situations to guide action and to region) have been adopted without modification,
monitor results. The experiences prove the im- while some other measures in the EMEA region
portance of someone responsible for the whole have been adjusted. Metrics owners, who are re-
process, not just as a reporter, but as a manager in sponsible for improving the performance measures
charge of concrete follow-ups and monitoring the that are part of their functional scorecards,
effects of actions, as well as being responsible for sometimes want to change the definition of certain
improving the PMS itself. This role is labeled the performance measures. The PM manager needs to
‘‘supply chain strategic planning and performance authorize changes in the definition, to make sure
manager’’ which we here call the ‘‘PM manager’’ that metrics remain consistent between various
for short. areas and with the global scorecard. The metrics
dictionary––although the template is not used ex-
4.4.2. Scorecard format plicitly––remains to be an important element for
The format of the scorecards has changed sig- achieving consistency. ‘‘I do not impose any re-
nificantly since the design project ended. This strictions regarding the format of the functional
confirms that the format is a fluid element for the scorecards, but close coordination regarding defi-
development and implementation of PMSs, nitions is needed’’ says the PM manager. The
while––as we will discuss in Section 4.4.3––the approach of starting with a detailed metrics dic-
individual metrics provide a much more stable tionary was also followed successfully in a more
foundation of the system. The design included recent initiative to further develop the customer
using six clusters and three hierarchical levels. It service functional scorecard. The experiences
took several months after the completion of the confirm the importance of stable metrics that are
design project to actually start the regular pro- coordinated at a detailed level.
duction of a scorecard. Using the originally de-
fined metrics, the format of the report did not 4.4.4. The role of information systems
follow the clustering or the hierarchical layering of Further developments of the PMS will be based
metrics. Actual target and the deviation were re- on the implementation of new information sys-
ported for each measure, and a 3-month average tems. The company will go live with new infor-
was added. Later on a graphical representation mation systems such as SAP, Siebel, and i2 by the
and a color code to signal whether the result was end of 2002, whereas in the current situation data
favorable or unfavorable compared to the target, are gathered in spreadsheets from a wide variety of
were added too. These lengthy reports were used in information systems to create the scorecards.
the functional one-on-one meetings. Experience These scorecards have a fixed format for the users,
showed that it was difficult to use these reports to and information about analysis, actions and re-
focus on exceptions and subsequently taking ac- sults is dispersed across documents and e-mails.
282 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286
The new systems should enable managers in dif- • the innovation processes that enable to firm to
ferent countries, business units, functional areas, remain successful in executing processes, deliv-
and at various managerial levels, to create reports ering on service objectives, resulting in sound
from the same data available in a common data- financial performance.
base. The company is testing software for report-
ing. This could lead to a more interactive use of Exploring such linkages and turning understand-
PMSs, where managers aggregate data, drilldown, ing into concrete PMSs that develop over time, is
make new cuts of the data, and see the information the most important theme in the literature. De-
on measurements, analysis and action much more velopment processes are discussed that focus on
clearly together. However, the implementation organizing discussions with different stakeholders
also creates a need for further coordination efforts to obtain their input, and on maintaining and
to make sure that the data required for the various updating the system. Conceptual issues involved in
reports are consistently available. At present, the setting up a PMS are also raised in the literature,
company is in the process of describing perfor- such as the clustering of performance metrics, the
mance metrics and data requirements, and decid- hierarchical structure of a PMS, or the implica-
ing on standards for performance metrics where tions of a PMS for managerial control. Many
necessary. conceptual questions are still not answered, for
The pending implementation of the new infor- example: What are the effects of clustering on de-
mation systems explains why the company is not cision-making processes [28]? What are the effects
investing many resources to further develop the of combining several measures into an overall
current PMS with the Excel spreadsheets. The score? Why do some firms rank, for example,
objective is to get experience with the current way quality improvement opportunities using some
of working, to make sure that these measurements quality measures denominated in financial terms
are possible with the new information systems, and (e.g. euros for rework, scrap, and warranty ex-
to start improving on aspects such as presentation penses), others in percentages or counts (e.g. defect
systems, clustering, aggregation, drilldowns and rates, or customer complaints), and some in survey
sorting the data. scale points (e.g. customer satisfaction indices)
[20]? What are suitable criteria for forming a
4.5. Method cluster (for example, a high correlation between
measures, or usersÕ perceptions of which measures
After having described the design process and belong to the same category)? Is it more important
the experiences during the first year, it is time to to have a broad variety in measures, or is it better
reflect on differences between the way the results to select a smaller set of measures that already
were obtained in the case study and those obtained have a high reliability and predictive validity [20]?
by other approaches. The literature discussed in Furthermore, while the literature discussed in
Section 2 emphasized the development of PMSs as Section 2 often approaches the development and
a design effort to translate strategy into actions, use of a PMS in an ‘‘neutral’’ way for improving
usually at the corporate level. The organizationÕs strategy implementation and operational pro-
financial objectives towards shareholders as well as cesses, it is not clear how the development and use
the strategic actions that have been formulated to of a PMS is shaped by the monitoring and control
achieve those objectives are made concrete in the side of it. ‘‘Political’’ issues such as the reward
measurements and the targets set for each mea- structure tied to the PMS, information asymmetry
sure. These actions deal with three aspects: between management levels and organizational
functions, slack building when setting targets be-
• the service and value the firm seeks to offer its come relevant to consider [34].
customers, Without diminishing the importance of the
• the internal processes that the firm needs to exe- conceptual issues mentioned above, and acknowl-
cute and improve, edging that we did not address these in the present
C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286 283
study, a main finding of our study is that other PMS. Working in parallel on four scorecards (for
elements proved to be also important, elements Operations, Transportation, Warehousing, and
that have not received much attention before. In Customer Service) in combination with periodical
our company––and this may actually apply to meetings, created a consistent framework for PM.
many more settings of developing and imple- This eventually resulted in a metric dictionary
menting PMSs in Operations––a key element in listing all metrics (including relevant attributes)
the process was the coordination effort involved in displayed by the four scorecards. Such a document
bringing PM to a higher level. There existed many is important for further development of an inte-
reports, at various organizational levels, both grated PMS.
inside and outside the Operations function. 2. Use a standard metric definition template
Also initiatives to improve PM were going on that includes all relevant metric attributes needed
throughout the organization, and the company to produce or reproduce metric values in a con-
was implementing new information systems and sistent way (see Table 3). Compile a metric dic-
needed to ensure the future provision of the right tionary from the current metric definitions to serve
data. All these circumstances placed coordination as a basis for development and as a reference for
in the center. communication with all parties involved. Con-
Building and sharing a detailed understanding fronting the metrics used with the organizational
of the definitions of performance metrics became objectives, we found gaps in the selection and
crucial to the development and implementation of identified superfluous information.
the PMS. While the literature and our initial ap- 3. Use a clustering that creates a basis for the
proach placed a lot of emphasis on the structure development of performance metrics and supports
and presentation of scorecards (such as clustering communication. The number of clusters as well as
in several perspectives) we found that such dis- the cluster criteria may vary from situation to
cussions––although useful for stimulating the situation.
thinking about the system––were not paramount 4. Feedback on the PMS is more useful if real
for moving forward. During the design effort the data is used. In the case of dummy data, users are
metrics dictionary grew to become the main less motivated to explore the possibilities of the
output and afterwards this has been used to ac- system and its shortcomings.
tually implement the ‘‘Operations Flash’’ without 5. Commitment also means that data specialists,
using the ideas about clusters, aggregation, and ICT staff and other employees able to deliver the
hierarchical structure. Again, we do not want to most reliable data about a process (including in-
suggest that structure and presentation are un- terpretation) are allowed to spend time on such an
important, but that the role of a performance initiative.
metrics dictionary has been underestimated in the 6. Postpone the selection of dedicated PM
literature. software until the basis of the PMS (the metric
dictionary) is mature. This avoids the purchase of
4.6. Some ‘‘lessons learned’’ expensive IT systems that might not bring the ex-
pected improvement in PM.
To conclude this section, we suggest some 7. The adoption and further development of
‘‘lessons learned’’, regarding the development and the PMS requires a PM manager who is a ac-
introduction of a PMS in Operations. We attempt cepted member of the management team that re-
to avoid too obvious suggestions, for example sponsible for the supply chain. The PM manager
about the need to involve all players throughout is responsible for the whole reporting and im-
the process, and we try to reflect on what we provement process, not just as an analyst or ac-
considered surprising compared to our starting countant, but as a manager in charge of concrete
point based on the literature on developing a PMS. follow-ups and monitoring the effects of actions,
1. A cross-functional alignment forum of man- as well as being responsible for improving the
agers and users delivers a basis for an integrated PMS itself.
284 C. Lohman et al. / European Journal of Operational Research 156 (2004) 267–286
[19] R.W. Hall, H.T. Johnson, P.B.B. Turney, Measuring up: [31] B.H. Maskell, Performance measurement for world class
Charting pathways to manufacturing excellence, Business manufacturing. A model for American companies, Pro-
One Irwin, Homewood, IL, 1990. ductivity Press, Cambridge, MA, 1991.
[20] C.D. Ittner, D. Larcker, Innovations in performance [32] D.M. McCutcheon, J.R. Meredith, Conducting case study
measurement: Trends and research implications, Journal research in operations management, Journal Of Operations
of Management Accounting Research 10 (1998) 205– Management 11 (3) (1993) 239–256.
238. [33] D. Medori, D. Steeple, A framework for auditing and
[21] H.T. Johnson, R.S. Kaplan, Relevance lost: The rise and enhancing performance measurement systems, Interna-
fall of management accounting, Harvard Business School tional Journal of Operations & Production Management
Press, Boston, MA, 1987. 20 (5) (2000) 520–533.
[22] R.S. Kaplan (Ed.), Measures for manufacturing excellence, [34] K. Merchant, Modern Management Control Systems,
Harvard Business School Press, Boston, MA, 1990. Prentice Hall, 1998.
[23] R.S. Kaplan, D.P. Norton, The balanced scorecard–– [35] A.J. Nanni, J.R. Dixon, T.E. Vollmann, Integrated
measures that drive performance, Harvard Business Re- performance measurement: Management accounting to
view 70 (January–February) (1992) 71–79. support the new manufacturing realities, Journal of Man-
[24] R.S. Kaplan, D.P. Norton, Putting the balanced scorecard agement Accounting Research 4 (1992) 1–19.
to work, Harvard Business Review 71 (September–Octo- [36] A.D. Neely, M. Gregory, K. Platts, Performance measure-
ber) (1993) 134–147. ment system design, International Journal of Operations &
[25] R.S. Kaplan, D.P. Norton, Using the balanced scorecard Production Management 15 (4) (1995) 80–116.
as a strategic management system, Harvard Business [37] A.D. Neely, A.H. Richards, J.F. Mills, K.W. Platts,
Review 74 (1) (1996) 75–87. M.C.S. Bourne, Designing performance measures: A
[26] R.S. Kaplan, D.P. Norton, Having trouble with your structured approach, International Journal of Operations
strategy? then map it, Harvard Business Review 78 (5) & Production Management 17 (11) (1997) 1131–1152.
(2000) 167–178. [38] N.G. Olve, J. Roy, M. Wetter, Performance drivers: A
[27] J. Lampel, H. Mintzberg, Customizing customization, practical guide to using the balanced scorecard, John Wiley
Sloan Management Review 37 (4) (1996) 21–29. & Sons, New York, 1999.
[28] M. Lipe, S. Salterio, A note on the judgmental effects of [39] R.J. Ormerod, Combining management consultancy and
the balanced scorecardÕs information organization, research, Omega, International Journal of Management
Accounting, Organizations and Society 27 (2002) 531– Science 24 (1) (1996) 1–12.
540. [40] M. Sanger, Supporting the balanced scorecard, Work
[29] F.A.B. Lohman, The effectiveness of management infor- Study 47 (6) (1998) 197–200.
mation, Ph.D. Thesis, Delft University, 1999. [41] G.D. Scudder, C.A. Hill, A review and classification of
[30] R.L. Lynch, K.F. Cross, Measure up!: Yardsticks for empirical research in operations management, Journal of
continuous improvement, Blackwell, Oxford, 1995. Operations Management 16 (1998) 91–101.