 Store  Log in Register

Explore Certifications Membership Standards & Publications Learning & Events Business Solutions 

Learning » Learning Library

Defining project success Share Share Tweet Share

a multilevel framework

ADVERTISEMENT
CONFERENCE PAPER ǀ Quality Management, Benefits Realization ǀ 16 July 2008
Bannerman, Paul L.

How to cite this article:

Bannerman, P. L. (2008). Defining project success: a multilevel framework. Paper presented at PMI® Research
Conference: Defining the Future of Project Management, Warsaw, Poland. Newtown Square, PA: Project
Management Institute.

NICTA, Australian Technology Park, Sydney, Australia

Introduction
ADVERTISEMENT
In the drive toward formalizing project management as a distinct discipline, there has been much
discussion on the nature and definition of project success, but no consensus has emerged. This
paper aims to contribute to the development of the field by synthesizing the seminal literature into a
multilevel framework of project success that has wide application in practice. The proposal is
supported and illustrated by reference to information systems (IS) development projects, but the
fundamental framework is relevant to most, if not all, project applications, including product
development, engineering, construction, aerospace, and defense.

Projects are discrete but multidimensional activities that serve as vehicles of change. They have a
checkered history of performance (especially IS projects). However, it is not easy to develop a
consistent view of the problem because of this multidimensionality. Empirical studies tend to use
different definitions of project success, making comparison difficult. In the literature, project success
variously refers to “on time, within budget, to specification” completion; success of the product
produced; or success in achieving the business objectives of the project. Also, these measures are Related Content
often contested, making it difficult to determine whether there is a problem at all (Sauer, Gemino, &
Reich, 2007). A further complication is that, like quality, success is perceptual and perceptions vary
ARTICLE ǀ Quality Management, Risk Management,
with the stakeholder's perspective and the passage of time since project completion. Despite these Benefits Realization, Stakeholder Engagement ǀ 1
April 2017
challenges, unlocking the problem of defining project success is critical for progressing project
management research and extending the knowledge base of this nascent discipline. PM Network
Extra Effort
Much work has already been done in researching project success but the main objective of that
By Hussain, Waqar ǀ Waqar Hussain, PMP,
research is often to unlock the drivers of project success rather than establish a common framework provides three approaches to take to
for determining whether a project is a success. The framework proposed in this paper comprises five complete projects delayed or complicated
levels of performance criteria that permit assessment of a project from multiple stakeholder by client-side problems: Improve clients'
perspectives at different times after project closeout. Three of these criteria (Levels 2 to 4) comprise capabilities; Engage during delays; and Pick
up the…
criteria commonly discussed in the research literature (project management success, product
success, and business success); another criterion (Level 5, strategic success) extends views that
suggest consideration of broader, future benefits; and a final criterion (Level 1, process success), not ARTICLE ǀ Quality Management, Complexity,
Benefits Realization ǀ 1 September 2016
discussed in the project management literature as a success criterion, supports learning and
development in project application domains. Based on this framework, project success is the highest PM Network
level achieved at any point of reflection. Full house
By Howlett, Sarah Protzman ǀ As the U.K.'s
The paper offers three main contributions. First, it formalizes two additional success criteria, one
largest casino resort, Resorts World
conceptually closer to the project action and the other further away (Levels 1 and 5). Second, it Birmingham caters to high rollers. But long
structures these and other extant criteria into a multilevel framework and assessment approach that before it opened in October 2015 in
has practical utility for determining project success. Third, it provides a rationale for using the Birmingham, England, the project team was
framework that contributes to overcoming the problems associated with defining project success by going all in the ensure…
aligning success determination to a common reference framework.
CONFERENCE PAPER ǀ Quality Management,
The paper is structured as follows. First, the literature is reviewed to position the problem, identify Benefits Realization ǀ 11 May 2015
success criteria, and uncover gaps in the research-practice nexus. Then the proposed multilevel
Beyond on-time and under-budget
project success framework is described. Application of the framework is then illustrated using
information systems case studies. The paper concludes with a discussion of the contribution and
directions for future research. By Burk, Susan M. ǀ This paper encourages
moving beyond assessing project execution
toward additionally examining whether or
Literature Review not a project has delivered its intended
outcome. It offers considerations for
Project management has a rich history (Cleland & Ireland, 2007; Kloppenborg & Opfer, 2002; Morris assessing the…
1994) but is still emerging as an independent discipline with its own theoretical foundation (Koskela &
Howell, 2002; Morris, 2002; Sauer & Reich, 2007; Söderlund, 2004). In particular, it lacks a common
ARTICLE ǀ Quality Management, Benefits
measure of project success and failure. Realization ǀ February 2015

Project Management Journal

Since a project is usually a means to an end rather than an end in itself, it is reasonable to want to
know if the project is successful, whatever the end might have been. Herein lies a difficulty. The The relationship between project
success of a project can be determined from the perspective of the means (the project itself) or the success and project efficiency
end (what it was intended or expected to accomplish) depending on the interests of the stakeholder.
Furthermore, regardless of means or ends, expectations of what the project was to achieve and By Serrador, Pedro | Turner, Rodney ǀ Many
perceptions of whether it achieved them often vary among stakeholders. This makes determination researchers have suggested that meeting
time, scope, and budget goals, sometimes
of project success highly contingent upon the expectations and perceptions of different
called 'project efficiency,' is not the
stakeholders, and when the assessment is made (de Wit, 1988). comprehensive measure of project success.
Broader measures of success…
Most projects have multiple stakeholders with different views on the project's purpose and different
expectations of what the project must achieve (Lyytinen & Hirschheim, 1987). These stakeholders
might include the people who originally identified the need for the project, those who fund the CONFERENCE PAPER ǀ Quality Management,
Benefits Realization ǀ 29 July 2014
project, those who stand to benefit from the project, the people who are impacted by the project and
its outputs, the project team members, and the people who have to oversee the project. Each has a Project benefit management
vested interest in the project's outcome, with different expectations and perceptions. By Zwikael, Ofer | Chih, Ying-Yi ǀ Realizing
benefits is an important criterion to evaluate
If we assume that a project is an end in itself, then its success can be determined at closeout stage. project performance. Hence, project benefit
However, if it is a means to an end, then its outcome can only be measured at some time after the management is essential to enhance project
formal project has completed. This permits the entry of other events and influences into the success. This paper focuses on the first
perception of project success that may not be a realistic reflection on the achievements of the step in the…

original project.

To unlock this issue of defining project success, we review how the problem has been approached in
the literature as input to proposing an integrated multilevel project success framework as a way
forward.

Project success has attracted much attention in the research and practice literature. Three main
streams are found. The first and dominant stream aims to identify the factors that practice suggests
are likely to contribute to project success, project failure, or project risk (e.g., Baker, Murphy, &
Fisher, 1988; Cooke-Davies, 2002; Pinto & Covin, 1989; Pinto & Slevin, 1988b; Schultz, Slevin, &
Pinto, 1987; Slevin & Pinto, 1986). Typically, this literature produces prescriptive lists of critical
success factors, failure factors or risk factors that project managers and governance bodies should
take account of to ensure a positive project outcome. The value of this research stream is that it
identifies important preconditions and drivers of project success, but it provides no explicit definition
of project success itself (although the factors identified may indirectly point to relevant criteria).

The second stream focuses on identifying other contingency variables that might impact project
outcomes or require specific management intervention to mitigate any potential negative effects.1
Some researchers call these variables “dimensions” of project success. They include project size
(Yourdan, 1997), project type (Pinto & Covin, 1989; Shenhar, Tishler, Dvir, Lipovetsky & Lechler, 2002),
life cycle stage (Pinto & Mantel, 1990), project management complexity (Shenhar & Wideman, 1996),
and strategic versus operational mindsets (Schultz, Slevin & Pinto, 1987; Shenhar, Poli & Lechler,
2000). The value of this research is that it identifies additional project variables that may have a
critical impact on project success, depending on the context of the project and how the variables are
managed. Again, however, this stream does not explicitly define measures of project success.

These two streams are often considered in conjunction with a third stream, whose main interest is in
defining the criteria by which a project is judged to be a success or a failure. The three streams are
interrelated in that the first two are concerned with how to achieve project success, while the third is
concerned with the measures against which success is judged. Knowing how success is defined is a
necessary precursor to determining where and how project effort should be focused to meet
performance goals; and knowing where to focus project management effort is guided by an
understanding of the drivers of project success and failure. Having a common definition of project
success also facilitates agreement on whether, in the face of disparate interests and perspectives,
success has been achieved. However, in the past, there has been an imbalance of attention in these
streams of research. Research and practice have tended to focus on “how to do it right” (the first two
streams) at the expense of reaching consensus on what “right” is (the third steam).

Some researchers suggest that success criteria should be project-specific and therefore determined
by stakeholders at the start of each project (e.g., Baccarini, 1999; Nelson, 2005; Turner, 2004;
Wateridge, 1998). This view has considerable merit because of the broad range of project types,
project objectives, and other variables that can contribute to project outcomes. However, there is
also a role for a common reference framework to enable project success to be discussed in a
uniform way and to provide a standard benchmark by which project outcomes can be compared
(Pinto & Slevin, 1988a), especially within the same discipline.

Several surveys of project success research already exist in the literature (e.g., Cooke-Davies, 2004;
Jugdev & Müller, 2005; Pinto, 2004; Shenhar, Dvir, & Levy, 1997; Wateridge, 1998). Furthermore,
Baccarini (1999) summarizes characteristics of project success criteria. The present review is not
intended to be exhaustive in uncovering either the range or depth of contribution of the literature to
this discussion. Rather, it focuses on identifying success criteria that the literature suggests would
present difficulties in practice if excluded from a definition of project success.

Project Management Success

The classic criterion from practice is a measure of the immediate performance of a project against its
main design parameters—schedule (time), budget (cost), scope, and/or quality—which the literature
tends to call a measure of project management success. This definition was already established in
the earliest discussion of projects in the management literature. Gaddis (1959) defined a project as
“an organization unit dedicated to the attainment of a goal—generally the successful completion of a
developmental product on time, within budget, and in conformance with predetermined performance
specifications” (p. 98). In the three-element form, this criterion is variously called the triple constraint,
iron triangle, or three-legged stool of project management. Other variants include all four elements as
the project diamond or four-legged stool. Scope is less clearly defined than time or cost, referring to
the extent to which the main deliverable was completed against specification or whether all intended
activities and phases of the project were completed. Quality is often assessed, post hoc, against
established industry or subjective criteria. The conventional approach is that an assessment of
performance is made in a post project review based on whether the project was completed “on time,
within budget and to specification.” If each was achieved within a narrow range of tolerance then the
project is deemed a success. This criterion is of particular interest to stakeholders with vested
interests in the project vehicle itself, such as the project manager, project team, and project
governance stakeholders.

This classic criterion remains the most widely used measure of project success. Its main value is in
offering a simple, direct measure of performance of a project and the project management expertise
applied to complete the project within the bounds of the most immediate design parameters (time,
cost, and scope). However, it has major limitations. Most critically, it focuses on the means rather
than the ends of the investment from the organizational perspective. It takes limited or no account
(depending on how scope is defined and measured) of whether the main project deliverable fulfilled
the purpose for which it was intended and whether the objectives of the project's investors were
achieved. For example, it is not unusual, especially in IS projects, for a project that is late, over
budget and/or under-delivered against specifications to be declared a success, because it still
delivered a benefit to the client/users and/or to the investing business. This suggests the need for
two additional success criteria: measures of project deliverable or product success and business
success.

Product Success

Drawing on the IS literature, several researchers implicitly argue that project success is a function of
the success of the information system produced by the project (e.g., Ballantine et al., 1996; Delone &
McLean, 1992, 2003; Seddon, 1997; Seddon, Staples, Patnayakuni, & Botwell, 1999). Completing
the main project deliverable “to scope” (specification) may not be a valid or sufficient measure of
project success if the deliverable is not also accepted and used by the intended client/end-user
and/or does not provide sufficient benefit to them. In the case of information system deliverables,
this success criterion might comprise measures of information quality, system quality, service quality,
intention to use, actual use, user satisfaction, and net benefits (DeLone & McLean, 2003). The same
concept, however, can be applied to buildings and other civil infrastructure, for example, produced
by construction and engineering projects (Baccarini, 1999). Clients/users have different interests and
expectations of a project to triple constraint stakeholders. They center on fitness for use and other
benefits associated with improvements in the nature and conditions of their work. A project can
succeed in delivering an information system “on time, within budget and to specification” but fail to
gain user acceptance or use of the system. It is well-recognized that this can occur, for example,
when a system specification lacks adequate user input to its definition and/or when user
requirements change due to evolving business circumstances.

This view of project success is fundamentally consistent with Pinto and Slevin's (1988a) seminal
model of project success and Kerzner's (2003) definition. Pinto and Slevin (1988a) modeled project
success as comprising two components: success of the project itself, as indicated by time, cost, and
performance subcomponents, and client success, as reflected by use, satisfaction, and effectiveness
of the project in benefiting intended users. Similarly, Kerzner (2003) defined project success as
completion within time, cost, and specifications (the traditional triple constraints), as well as with
minimum or mutually agreed scope changes and acceptance by the client/user (what I have called
product success). Kerzner added two additional components: completion without disturbing the
main workflow of the organization and without changing the corporate culture. The intent of these
components is not to argue that projects should not be vehicles of change within the organization
but an acknowledgement that projects execute within an existing operational organizational context
with established values and norms of behavior. This is consistent with the view of a project as a
discrete change activity within an organization.

Business Success

In the case of the second additional criterion, a measure of business success, de Wit (1988) argued
for distinguishing between project success and project management success. The latter is
determined by the triple constraints of time, cost, and quality, but the former is a measure of the
degree to which the project objectives are met and benefits accrue to the investing organization.
Other researchers refer to this additional criterion as the business or organizational objectives
criterion (e.g., Ballantine et al., 1996; Shenhar, Dvir, Levy, & Maltz, 2001). Describing these objectives
as business or organizational objectives rather than project objectives begins to resolve the problem
raised by de Wit (1988) of multiple stakeholder objectives relating to a project. Simplistically, project
objectives relate to the goals in the project plan while business or organizational objectives relate to
the goals in the business plan.

Taking information systems, ultimately, businesses do not invest per se in a new computer system
for the right system to be installed on time, within budget, to specification, and the satisfaction of
users. Instead, they aim to solve a particular business problem (albeit in a timely, cost-efficient, and
effective manner). If the project does not deliver an acceptable solution to that problem then
investment stakeholders are likely to view the project as a failure. Naturally, the business success
criterion also permits the perverse possibility encountered in practice of a project failing on project
management and/or project deliverable criteria but still achieving business objectives in some
acceptable way and, therefore, being considered a success. This reinforces the counterintuitive view
that project management success and even project deliverable success are neither necessary nor
sufficient for project success.

Not all researchers support this three-tiered view of project success, as presented so far. For
example, discussing information systems failure (rather than success), Lyytinen and Hirschheim
(1987) argued for the notion of expectation failure (“the inability of an IS to meet a specific
stakeholder group's expectations,” p. 263) in favor of what they called process failure (the IS is not
designed within time and cost constraints), interaction failure (the IS is not used), and
correspondence failure (the IS does not match goals). Stakeholders are defined as “all those
claimants inside and outside the organization who have a vested interest in the problem and its
solution” (p. 261). Their view is that failure (success) is the embodiment of a perceived situation, and
stakeholder perception/expectation is a superset of the other three criteria. This view is inconsistent
with a definition of project success based on a single criterion, but it is consistent with a multilevel
project success framework that permits multiple stakeholder interests at different levels of perception
and at different timeframes as proposed in this paper. It is also consistent with de Wit's (1988) view
that stakeholders have many different objectives of a project. This leads us to a fourth success
criterion.

Strategic Success

De Wit's (1988) concerns about multiple stakeholder objectives are further partially addressed by
criteria that assess project success beyond direct organizational benefits. These include benefits that
favorably position the organization for future opportunities (Shenhar et al., 2001), and benefits that
accrue to the stakeholder community beyond the investing organization (Atkinson, 1999). In the
proposed framework, these considerations are generalized as a strategic success criterion. This
criterion represents the highest level of benefit achieved by a project, despite the possibility of
failures against lower level criteria, as recognized by external stakeholders, such as investors,
industry peers, competitors, or the general public, dependent upon the nature of the project.

Take an example from engineering and construction, the Sydney Opera House. The building was
completed 10 years late, nearly 15 times over budget and significantly functionally constrained (its
small opera stage and buried orchestra pit are well-known), yet it has served Sydney's performing
arts patrons successfully for 35 years, has been a boon to its owners, and the appeal of its
distinctive architecture has attracted international attention, including recognition in 2007 as a
UNESCO World Heritage Site. Typically, few projects achieve strategic success. However, this
criterion enables the perceptions of a broader community of stakeholders than those in the investing
organization to be represented, and a broader range of benefits than might have been intended. It is
the ultimate confirmation that a project delivered an outstanding end result.

In addition to engineering and construction, this criterion is particularly relevant to information

systems projects. In the 1980s, the concept of strategic information systems emerged around the
use of information systems to gain competitive advantage and other strategic benefits (Porter &
Millar, 1985). While a number of strategic information systems are identified in the literature (e.g.,
Ciborra & Jelassi, 1994), it is also recognized that achieving strategic information system status is
not simply a matter of design or intent (Emery, 1990). Its success depends on its effectiveness in the
marketplace and is dependent on the perceptions of competitors and other industry stakeholders
external to the investing firm.

Process Success

A final project success criterion is not recognized in extant project management literature. It
responds to the need to consider technical and managerial processes associated with project
management that are important at different times throughout the project life cycle. Consequently, this
is a lower level criterion than the project management criterion discussed above. Consistent with the
aims of quality management (such as Deming's Quality Management System, Total Quality
Management, Motorola's Six Sigma, Lean, and ISO 9000 standards), this criterion provides a basis
of focus on continuous improvement of critical discipline-specific in-project processes.

In the case of information systems, for example, the systems and software engineering literature has
a strong interest in the processes that underpin successful information systems development and
deployment. For example, software process improvement (SPI) is a key component of software
quality accreditation (Chrissis, Konrad, & Shrum, 2007). Key related processes include software
development methodologies, configuration management, risk management, change management
and quality control within the context of project and technical governance mechanisms. Most post-
project reviews include consideration of these processes in determining project management
success. Reviews typically consider whether the right processes were chosen and effectively applied
when needed, and whether they were appropriately aligned and integrated to facilitate achieving the
objectives of the project.

However, there is no explicit criterion in the project success literature to focus attention and learning
on these critical in-project processes. The absence of such a criterion makes it difficult, for example,
for a stakeholder outside of the project to know whether a project was late because of poor schedule
management or some other embedded process within the project. A lower level criterion would
facilitate a finer grained examination of performance that could directly support incremental learning
and improvement of contributing processes.

Project disciplines other than information systems (such as product development, construction,
aerospace, etc.) have their own sets of critical in-project processes that could similarly be assessed
independently of their net effect on the overall project management result with a success measure at
this level of analysis.

The above literature-based analysis has focused mainly on identifying different project stakeholder
perspectives. However, these perspectives also correlate with time; that is, with when the
assessment is made. For example, if the assessment is made immediately after project closeout,
there is little more in extant project success definitions than the traditional triple constraint criterion
upon which to base the determination. After the passage of time, however, assessment becomes
possible against the criteria of the other interests and perspectives. Pinto and Slevin (1988a), for
example, argued that

there are definite benefits involved in waiting until after the project has been transferred to
the clients for whom it is intended before assessing project success. By waiting until the
project is up and functional, we are better able to understand the impact of the external
organizational factors, such as value and organizational effectiveness (impact). (p. 70)

Viewed alternatively as a financial investment, a “successful” project may show increasing returns on
investment over time, so perceived value and success may increase with time. As with any
investment, the derived value (return on investment) can be assessed at multiple points in time after
the initial investment and at multiple levels of analysis (that is, from different perspectives).

This contrasts with the current tendency to determine project performance at only one point in time
(hence the argument to delay the assessment until the most appropriate time). This latter approach
creates the dilemma that if project performance is determined at one point in time, which point is
“best”? Determining this point has been the preoccupation of much of the extant literature on project
success. The proposed multilevel framework, which we now consider, overcomes this dilemma.

Multilevel Project Success Framework

An alternative approach to the problem of defining project success is a framework that enables
success to be determined at key milestones at different times after project closeout and from
different stakeholder perspectives (Figure 1). Key milestones in this spectrum relate to the project
itself (the processes used and their effectiveness in delivery the project within the major design
constraints), the product or main deliverable produced by the project (its fit to specifications and
purpose as well as acceptance and use), and the organizational benefits returned from the
investment (achievement of business objectives and the generation of strategic value). These
milestones represent five levels at which project-related performance can be formally or informally
assessed. Levels 2 to 4 reflect criteria commonly found in the literature. Level 5 is implied by some
researchers but made explicit in this framework, while Level 1 incorporates a measure of technical
performance found in disciplinary domains to promote learning and development at the tactical level
in the design and execution of projects.

Figure 1. Five Levels of Project Success

This approach enables success to be determined and periodically re-determined as benefits accrue
from the project over time. It also enables stakeholders to progressively map success to perceptions
of higher derived value from the project as benefits accrue. Based on this framework, project
success is defined by the highest level of benefit achieved by the project at any point of reflection.
This makes it possible, even non-controversial, for a project to fail at a lower level of assessment but
still succeed at a higher level of perceived return from the project.

Each level is briefly described, following, and detailed in Table 1.

Level 1 – Process success. Every project discipline has generic and project-specific best practices
that are critical to successfully completing a project. Even generic processes such as project
management and risk management have discipline- and domain-specific best practices. Others will
be exclusive to the discipline. Determination of success at this level considers the appropriateness of
the processes used, their alignment with the project's purpose, and their integration and
effectiveness in contributing to the project outcomes. As with the other levels, analysis here provides
feedback to the project team and organization for learning and improvement for subsequent
projects.

Level 2 – Project management success. This is the traditional criterion of project success,
determined on closeout against key project design parameters such as the project schedule, budget
and some performance expectations such as completing all planned stages and activities. Note that
scope at this level refers to project scope, not product scope (the latter is a component of the next
level).

Level 3 – Product success. This level considers the success of the major deliverable from the
project. Clearly, what this is will vary with the project discipline and the specific project (e.g., an
information system, building, submarine, road, or some form of service deliverable). It includes
measures relating to the deliverable itself (such as its match to specifications, requirements, and
quality expectations) and to client/user satisfaction (such as product acceptance, use, and
effectiveness).

Level 4 – Business success. Success at this level is accounted as accrual of positive net benefits
to the organization from the project. It may also include an assessment of the organizational
contribution to the project's outcome. Consequently, measures will typically include the degree to
which the project met the goals and objectives that motivated the investment approval (which are
usually specified in the business plan), and whether the expected benefits were realized. They may
also include consideration of the effectiveness and contribution of corporate governance to the
project. Finally, assessing net benefits will also include any unintended benefits and negative impacts
that arose from the investment.

Level 5 – Strategic success. At this level, organizational benefits are assessed by external
stakeholders such as investors, competitors, industry analysts, or regulators, rather than company
insiders. Success at this level derives from net improvements in industry position, business growth
and development, competitive advantage, and/or other strategic gain. Strategic success may be
planned or emergent.

The empirical indicators in Table 1 are adapted and extended from Shenhar et al. (2001). They are
generic pointers to project-specific criteria.

Table 1. A MultiLevel Framework of Project Success

Level Success Criterion Description Empirical Indicators

1 Process Discipline-specific technical and Technical and managerial processes

managerial processes, methods, were:
tools, and techniques employed to Appropriately chosen for
achieve the project objectives. the purpose
Aligned with the project
objectives
Integrated with each other
(as appropriate)
Effectively implemented

2 Project The project design parameters or Schedule met

Management objectives. Here “scope” refers to
Budget not exceeded
the intended scope of the project
(e.g., to specify, build, test, and Project scope achieved
implement a new system), not the
scope of specifications of the main
project deliverable.

3 Product The main deliverable(s) from the

Specifications met
project. The nature of the
deliverable(s) will be discipline- Requirements met
specific. For example, it might be a
Client/user expectations
product, system, building, bridge,
airplane, rocket, or a service of met
some kind. Client/user acceptance
Product/system used
Client/user satisfied
Client/user benefits
realized

4 Business The business objectives that

Objectives met
motivated the investment. That is,
what the business wanted to Business case validated
achieve from the investment.
Business benefits realized

5 Strategic Business expansion or other

Business development
strategic advantage gained from the
enabled
project investment, either sought or
emergent. External
stakeholder/competitor
recognition
Competitive response
generated

In using the multilevel framework, five basic “rules” apply:

1. Project success may be determined at successively higher levels as time passes from project closeout.

2. A project may succeed at a level but “fail” (or, more correctly, not succeed) at one or more subordinate levels.
For example, it is possible to succeed at Level 4 (business success) but not be successful at one or more of the
lover levels (Levels 1 to 3). That is, project success at one level implies nothing about the project's performance
at other levels.

3. Project success is determined by the highest level achieved at any point in time. For example, if a project is
determined to have succeeded at Level 4 (business success), the project's success status is that it is/was
successful at Level 4, regardless of its performance at lower levels.

4. Project success eventually becomes historic, as the work of the project and/or its major deliverables becomes
obsolete. The timeframe during which this occurs will vary with the project discipline (a building may last 100
years or more, but a computer system might be replaced within five years). The historic success of the project
remains the highest level achieved during its operational life.

5. Project “failure” at one or more levels is an indicator of the possible need for organizational learning and
development at that level, to avoid repetition in subsequent projects.

The framework comprises generic criteria that should be applicable to most projects from most
disciplines. While it offers improvements on extant approaches to defining project success, there is
no panacea for differences of opinion arising between individuals on whether a project is successful
or not. Success at a particular level may still be a matter of majority opinion. However, consistent
with advice from the literature (discussed in the previous section), this problem can be mitigated by
each project defining specific performance criteria within each success level (as relevant to the
project), in the business case and project plan for later assessment of performance outcomes. If
these project-specific performance criteria are defined in a manner that is consistent with the
success levels as described in Table 1, meaningful comparison of project outcomes can still be
undertaken.

Information Systems Project Case Illustrations

The following information systems examples illustrate project success and failure using the proposed
framework.

Sydney Water's CIBS Project – A Failure

As reported in NSW-AG (2003), in June 2000, Sydney Water, a government owned utility company,
let contracts to build and implement a Customer Information and Billing System (CIBS). The project
was intended to improve service to customers, fill gaps in existing information systems, and provide
business efficiencies. The new system was to integrate with 12 existing internal systems and 60
external party interfaces. The system was expected to be operational by February 2002, at a cost of
$38.2 million (AUD). Due to the board's concerns about delays, excessive costs (the estimate had
blown out to $135.1 million (AUD)), and failure to reach acceptable standards, the project was
terminated on 30 October 2002 at a loss of $61 million (AUD). A government audit review of the
project found deficiencies in:

Project governance by management and the board. Management did not keep the board
informed of important project issues and the board did not oversee the project as effectively as it
might have, given its significance and complexity. For example, the project was approved before
an IT architecture was defined. After an architecture was put in place, CIBS was found to fail 19
of 20 architectural requirements.
Project specification, project management, and interface with users. Project planning and
specifications were inadequate, resulting in a stream of change requests, and communication
with users was poor. An integrated project plan was not maintained throughout the project and
the business case was not maintained. Also, testing was poorly managed.
Contractual arrangements. Contract administration was deficient. For example, the contract was
weaker than the request for tender document. Furthermore, one contract variation transferred
significant responsibilities and risk from the contractor back to Sydney Water.
Risk management. Risk management was ineffective within the project and at the organizational
level. There was a view that outsourcing the project transferred the risk to the contractor.
While this project clearly failed at Level 2 (because it did not complete), the audit report indicates that
there were significant failures at Level 1 that contributed to the delays and cost overruns, resulting in
loss of confidence in the project and its eventual termination. A number of recommendations for
improvement were made for future projects.

St. Henry's Hospital's BMS Project – Success at Level 1 (Almost)

This project also “failed” in that it did not complete, but failure at the process level was less obvious.
St. Henry's is a pseudonym for a government-owned hospital that initiated a half-million dollar (AUD)
project in 2002 to build a web-based Bed Management System (BMS). The scope of the project was
to specify, design, build, and implement an information system to support real-time bed bookings,
scheduling, bed status reporting, and bed data publication to key stakeholders (including ambulance
services and the emergency departments of other hospitals in the region). It was also to integrate
with the hospital's patient administration and clinical systems.

Six months into the project, the steering committee became aware of plans by the organization that
administered all government-owned hospitals in that jurisdiction to implement a new patient
administration system in each of its 250 hospitals and a centralized electronic health record
database system. This created unexpected dependencies for the local project initiative at St.
Henry's. The BMS project was halted for a protracted investigation of the options available to St.
Henry's.

The jurisdiction-wide system rollout would not provide the functionality St. Henry's wanted for its
BMS. However, for contractual reasons (with the system providers), St. Henry's could not proceed
with developing extensions to the new systems ahead of completion of the roll-out. Consequently,
the BMS project was eventually discontinued (from the hospital's perspective, the project was
“cancelled” rather than “failed”).

Ironically, in terms of Level 1 processes, the project was well managed by a contracted project
management professional up to the time of its termination. A detailed business case, risk analysis,
project plan, functional requirements, architectural design, database design, and interface
specifications were developed and maintained, and a hospital process review was conducted to
produce an efficient practice model for the new system. Furthermore, the project foresaw the
possibility of its own demise. The biggest risk identified for the project was that it might encounter
dependencies on other system projects over which it had no control. However, as in surgery, the
notion that “the operation was a success but the patient died” carries little comfort for “patient”
stakeholders.

More typically, a Level 1 success might be determined at project closeout for a project that failed one
or more of the Level 2 measures (time, cost, or project scope) for reasons other than process failure.

CompuSys' CONFIG System Project – Success at Level 2

Markus and Keil (1994) recounted a classic case of a textbook system development project that
produced a system that users would not use. CompuSys, a computer company, built CONFIG, an
expert system, to check the configuration specifications of new computer systems, prepared by
sales representatives, before the sale. This would avoid losses from supplying necessary
components that were omitted from the quotation. The system was a technical success in terms of
Levels 1 and 2 of the framework, and produced better configurations than the average sales
representative, but the sales force would not use the system. That is, it failed at Level 3.
 Having determined that the problem must have been human factors, the system developers
embarked on a major redevelopment to improve the user interface. This time, the users agreed that
the second version of CONFIG was far superior to the first but they still would not use it. On
investigation, it was found that sales reps had no motivation to use the system because they were
measured on sales, not occasional losses due to missing cables. Also, the system made it harder
rather than easier to generate quotations because CONFIG was not integrated with PQS, the
Product-pricing and Quotations-generation System. Markus and Keil (1994) concluded that this
outcome was the result of “process suboptimization through subprocess optimization.” That is, a
myopic focus on the configuration process blinded CompuSys to the context of the larger sales
quotation process.

Admin's GovNet Project – Success at Level 3

In another example of technical myopia, in the second half of the 1990s, Admin (a pseudonym for a
government department) responded to a new whole-of-government policy to extend transaction
services to the public via a web-based application I will call GovNet. The services were provided by
interfacing new web applications to Admin's existing in-house online transaction processing system.
The system presented several technical challenges because web application development platforms
were evolving and still immature at that time, and the main system had data management controls
that the add-on applications could easily circumvent if great care was not taken. Despite these
challenges, the project was completed successfully, gaining strong acceptance and use by early
adopters of web-based services.

It quickly became apparent, however, that Admin had not thought through the operational
implications of such a relatively easy extension to its transaction system (technical challenges aside).
Naturally, users expected to be able to access the systems on the Internet 24 x 7, but the base
transaction system, which included the database, was designed to operate only during normal
business hours. Out-of-hours time was dedicated to batch updates, batch printing and reporting,
extracts for dependent systems, backups, and other system “housekeeping.” Also, Admin had
insufficient staff and no budget to run multiple help desk shifts to provide telephone support to
online users at night. At best, availability of the web applications could only be extended a few hours
either side of the normal business day, and without help desk support. This resulted in considerable
embarrassment for the department and great pressure to resolve the problem, which could not be
done in any meaningful practical way within the design of the existing base system. Despite
achieving the immediate goal of compliance with government policy, the net benefits of the project to
the organization at Level 4 were negative. Consequently, the project's success was capped at Level
3 (given acceptance by users of the limited system availability).

DMV's LARS Project – Success at Level 4

In 1990, an Australian DMV (Department of Motor Vehicles) initiated a project to develop and
implement a new Licensing And Registration System (LARS) to replace its legacy batch processing
mainframe system. The system was to be completed in 18 months at a cost of $29 million (AUD).
The system was to be developed in-house using contract developers and new, high-productivity
development technologies that would automatically generate programs from design specifications.
The project was highly innovative on several fronts. The system design was innovative and required
integration of new technologies that had been previously untried. The development method was also
new and unproven for the chosen system platform. Furthermore, the project was well organized into
five teams for development (the largest), systems environment, data conversion, change
management, and procedures.

From the beginning, major problems confronted each team. Delays ensued, compromises had to be
made, due dates were frequently extended, and eventually the implementation had to be split into
two phases with scaled-down deliverables to get a system implemented. A licensing module was
implemented nine months late, followed by the registration component a year later. On
implementation, as the LARS general manager expressed it, “All hell broke loose.” The system was
slow and regularly locked up, customer data records were inaccurate or could not be found at all,
standard policies were enforced inappropriately, and customer queues became intolerably long at
the DMV office counters. Public outcry ensued. However, in the following months and years, the
problems were progressively and fully resolved. Some estimates put the final cost of LARS at $80
million (AUD).

Curiously, while the project struggled with technical and user problems, it did achieve the major
business objectives set for it. It replaced an unsupportable legacy system and saved the organization
$20 million (AUD) per year in costs associated with the old way of administering driver licenses and
vehicle registrations. Indeed, it was considered such a business level success that the Department
won an industry Award for Excellence for the new system. The project succeeded at Level 4 despite
failing against the criteria of all lower levels.

American Airlines' SABRE System Project – Success at Level 5

A final example is another classic case from the IS literature. American Airlines is widely recognized
for developing SABRE, the world's leading computerized airline reservation system. The
development of SABRE is well chronicled (Copeland & McKenney, 1988; Copeland, Mason, &
McKenney, 1995). Through most of its history, SABRE dominated the US airline industry,
successively evolving from a computer-based airline reservations system to a passenger service
system, a sales distribution system, and an electronic travel supermarket, setting the benchmark for
other airlines to adopt or compete against. Indeed, it is reputed that SABRE was so successful that
the system became more profitable for American Airlines than the airline business itself.

Discussion

This paper aims to contribute to the formalization of project management as a theoretical discipline
by synthesizing extant research into a multilevel framework that offers the potential to unite disparate
perspectives on projects in a consistent and uniform way of determining outcomes. Reaching
consensus on defining project success is likely to smooth the path to a better understanding of the
responsibilities and boundaries of project management as an enabler of change within organizations
and the wider stakeholder community. While consensus is not necessary for theoretical progress
(indeed, the reverse is usually true), a reference framework can aid in providing clarity of focus by
mapping the high level footprint of the discipline at a particular stage of its development, and depth
of focus by highlighting key dimensions of projects that stakeholders view as being important to
them. Much of the current focus of project management research and practice is limited to
prescriptions for achieving success at Level 2, yet project stakeholders indicate broader interests in
projects as a vehicle for delivering change and new capabilities.

The paper has limitations. The proposal is conceptual, developed from a review of the literature and
illustrated empirically with six case examples. The relevance and completeness of the criteria in the
framework and the framework's practical utility must be tested by application to many projects over
time. Also, generalization of the framework to apply to other project disciplines requires validation in
practice. Ultimately, as with projects, it is likely to be project stakeholders who determine the value of
the framework.

The paper has implications for research and practice. The paper extends the current body of
knowledge on project management by providing a way to integrate current thinking on the definition
of project success. It also extends this thinking by explicating two additional project success criteria.
This integration points to a broader scope of relevance of projects than that defined by the traditional
project life cycle. The perceptions of stakeholders suggest that they value outcomes attributable to
the original project well beyond closeout. Consequently, future research has the opportunity to
investigate the role of projects and project management in satisfying stakeholder expectations and
perceptions beyond the narrow confines of “on time, within budget, and to specification”
performance.

For practice, the framework enables different project interests, perspectives and outcomes to be
normalized in parallel rather than in competition with each other. It uncouples project management
success (Levels 1 and 2) from the post-project benefits that subsequently flow to stakeholders
(Levels 3 to 5), enabling acknowledgement of the highest value derived, regardless of tactical
performance. A practical danger here, however, is that success at a high level is not misinterpreted
as excellence at all subordinate levels. Each level has meaning only within that level. “Failure” at a
lower level may be equally, if not more, important than the success if that failure points to a need for
improvement. For example, that the DMV's LARS system ultimately “succeeded” in achieving the
required business benefits may obscure the failures at Levels 1 to 3 for this project (even though the
net financial benefits were diminished by up to $50 million (AUD) in additional development costs).
The critical importance of those failures comes into relief for the next major development project. The
outcome of each successive project is zero-based, save for the experiential learning in managing
projects that the organization accumulates over time.

Conclusion

This paper has argued that project management as a formal discipline would benefit from a unifying
representation of project success. Such a contribution would help focus research attention on
aspects of projects and project management that are important to stakeholders. It would also
facilitate communication and comparison through a common language of project success.
Acknowledging the realities of practice, this reference framework would have to accommodate the
multiple expectations and perceptions that arise from different stakeholder perspectives at different
positions of interest and time. The multilevel framework of project success developed in this paper
aims to fulfill these objectives.

Organizations in all disciplines are increasingly finding that project-based work, supported by project
management best practice, offer considerable utility over traditional functional designs in managing
discrete activities. As the discipline of project management matures, this trend is likely to continue.
Therefore, it is important that the discipline continues to develop and periodically formalize its
position (in agile software development, this is called refactoring). The challenge of this paper for
practice is to recognize and celebrate success in a way that is meaningful to others, aided by the
proposed framework, while progressively improving project management capabilities. The challenge
for research is how to respond to the wider community of project stakeholders whose interests lie
beyond the project life cycle as currently conceived.

References

Atkinson, R. (1999). Project management: Cost, time and quality, two best guesses and a
phenomenon, its time to accept other success criteria. International Journal of Project
Management, 17(6), 337-342.

Baccarini, D. (1999). The Logical Framework Method for defining project success. Project
Management Journal, 30(4), 25-32.

Baker, B. N., Murphy, D. C., & Fisher, D. (1988). Factors affecting project success. In D. I. Cleland &
W. R. King (Eds.), Project management handbook (2nd ed.), pp. 902-919. New York: Van Nostrand
Reinhold.

Ballantine, J., Bonner, M., Levy, M., Martin, A., Munro, I., & Powell, P. L. (1996). The 3-D model of
information systems success: The search for the dependent variable continues. Information
Resources Management Journal, 9(4), 5-13.

Chrissis, M. B., Konrad, M., & Shrum, S. (2007). CMMI: Guidelines for process integration and
product improvement (2nd ed.). Boston: Addison-Wesley.

Ciborra, C. & Jelassi, T. (Eds.). (1994). Strategic information systems: A European perspective.
Chichester, UK: John Wiley & Sons.

Cleland, D. I. & Ireland, L. R. (2007). Project management: Strategic design and implementation
(5th ed.). New York: McGraw-Hill.

Cooke-Davies, T. (2002). The “real” success factors on projects. International Journal of Project
Management, 20(3), 185-190.

Cooke-Davies, T. (2004). Project success. In P. W. G. Morris & J. K. Pinto (Eds.). The Wiley guide to
managing projects, pp. 99-122. Hoboken, NJ: John Wiley & Sons.

Copeland, D. G. & McKenney, J. L. (1988). Airline reservations systems: Lessons from history. MIS
Quarterly, 12(3), 353-370.

Copeland, D. G., Mason, R. O., & McKenney, J. L. (1995). Sabre: The development of information-
based competence and execution of information-based competition. IEEE Annals of the History of
Computing, 17(3), 30-56.

DeLone, W. H. & McLean, E. R. (1992). Information systems success: The quest for the dependent
variable. Information Systems Research, 3(1), 60-95.

DeLone, W. H. & McLean, E. R. (2003). The DeLone and McLean model of information systems
success: A ten-year update. Journal of Management Information Systems, 19(4), 9-30.

De Wit, A. (1988). Measurement of project success. International Journal of Project Management,

6(3), 164-170.

Emery, J. C. (1990). Misconceptions about strategic information systems. MIS Quarterly, 14(2), vii-
viii.

Gaddis, P. O. (1959). The project manager. Harvard Business Review, 37(3), 89-97.

Jugdev, K. & Müller, R. (2005). A retrospective look at our evolving understanding of project success.
Project Management Journal, 36(4), 19-31.

Kerzner, H. (2003). Project management: A systems approach to planning, scheduling, and

controlling (8th ed.). Hoboken, NJ: John Wiley & Sons.

Kloppenborg, T. J. & Opfer W. A. (2002). The current state of project management research: Trends,
interpretations, and prediction. Project Management Journal, 33(2), 5-18.

Koskela, L. & Howell, G. (2002). The underlying theory of project management is obsolete.
Proceedings of the PMI Research Conference, 293-302.

Lyytinen, K. & Hirschheim, R. (1987). Information systems failures: A survey and classification of the
empirical literature. Oxford Surveys in Information Technology, 4, 257-309.

Markus, M. L. & Keil, M. (1994). If we build it they will come: Designing information systems that
users want to use. Sloan Management Review, 35(4), 11-25.

Morris, P. W. G. (1994). The management of projects. London: Thomas Telford.

Morris, P. W. G. (2002). Science, objective knowledge, and the theory of project management.
Proceedings of the Institution of Civil Engineers – Civil Engineering, 150(2), 82-90.

Nelson, R. R. (2005). Project retrospectives: Evaluating project success, failure, and everything in
between. MIS Quarterly Executive, 4(3), 361-371.

NSW-AG, (2003). Review of Sydney Water's Customer Information Billing System. NSW Auditor
General's Report to Parliament, 1. Retrieved June 2005 from
http://www.audit.nsw.gov.au/agrep03v1/SpecialRevSydneyWaterCIBS.pdf

Pinto, J. K. (2004). The elements of project success. In D. I. Cleland (Ed.), Field guide to project
management (2nd ed.), pp. 14-27. Hokoben, NJ: John Wiley & Sons.

Pinto, J. K. & Covin, J. C. (1989). Critical factors in project implementation: A comparison of

construction and R&D projects. Technovation, 9(1), 49-62.

Pinto, J. K. & Mantel Jr., S. J. (1990). The causes of project failure. IEEE Transactions on
Engineering Management, 37(4), 269-276.

Pinto, J. K. & Slevin, D. P. (1988a). Project success: Definitions and measurement techniques. Project
Management Journal, 19(1), 67-72.

Pinto, J. K. & Slevin, D. P. (1988b). Critical success factors across the project life cycle. Project
Management Journal, 19(3), 67-74.

Porter, M. E. & Millar, V. E. (1985). How information gives you competitive advantage. Harvard
Business Review, 63(4), 149-160.

Sauer, C. & Reich, B. H. (2007). What do we want from a theory of project management? A response
to Rodney Turner. International Journal of Project Management, 25(1), 1-2.

Sauer, C., Gemino, A., & Reich, B. H. (2007). The impact of size and volatility on IT project
performance. Communications of the ACM, 50(11), 79-84.

Schultz, R. L., Slevin, D. P., & Pinto, J. K. (1987). Strategy and tactics in a process model of project
implementation. Interfaces, 17(3), 34-46.

Seddon, P. B. (1997). A respecification and extension of the Delone and McLean model of IS
success. Information Systems Research, 8(3), 240-253.

Seddon, P. B., Staples, S., Patnayakuni, R., & Bowtell, M. (1999). Dimensions of information systems
success. Communications of the AIS, 2, 2-60.

Shenhar, A. J. (2001). One size does not fit all projects: Exploring classical contingency domains.
Management Science, 47(3), 394-414.

Shenhar, A. J., Dvir, D., & Levy, O. (1997). Mapping the dimensions of project success. Project
Management Journal, 28(2), 5-13.

Shenhar, A. J., Dvir, D., Levy, O., & Maltz, A.C. (2001). Project success: A multidimensional strategic
concept. Long Range Planning, 34(6), 699-725.

Shenhar, A. J., Poli, M., & Lechler, T. (2000). A new framework for strategic project management. In T.
Khalil (Ed.), Management of technology VIII. University of Miami, Miami, FL.

Shenhar, A. J., Tishler, A., Dvir, D., Lipovetsky, S., & Lechler, T. (2002). Refining the search for project
success factors: A multivariate, typological approach. R&D Management, 32(2), 111-126.

Shenhar, A. J. & Wideman, R. M. (1996). Project management: From genesis to content to

classification. INFORMS Conference, Washington DC, May, downloaded from
http://www.maxwideman.com/papers/genesis/genesis.pdf

Slevin, D. P. & Pinto, J. K. (1986). The project implementation profile: New tool for project managers.
Project Management Journal, 17(4), 57-70.

Söderlund, J. (2004). Building theories of project management: Past research, questions for the
future. International Journal of Project Management, 22(3), 183-191.

Turner, J. R. (2004). Five necessary conditions for project success, International Journal of Project
Management. 22(5), 349-350.

Wateridge, J. (1998). How can IS/IT projects be measured for success? International Journal of
Project Management, 16(1), 59-63.

Yourdon, E. (1997). Death march: The complete software developer's guide to surviving “mission
impossible” projects. Upper Saddle River, NJ: Prentice Hall PTR.

_____________________________

1 I use contingency variables in the sense of organizational contingency theory, such as discussed by Shenhar
(2001), not in the common sense, in the context of project planning, of provisioning for uncertainty. Classical
contingency theory argues that project performance is contingent upon congruence between a project's structure
and its environment. That is, a project must be designed and managed according to its contextual circumstance.
The variables identified by this research stream point to contexts in which different project designs and
management actions may be necessary for the project to succeed.

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this
material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.
© 2008 Project Management Institute

ADVERTISEMENT

Quick Links Certifications Community Membership Organization

Webinars  Project Management Professional Latest from the Community  Membership Overview About Us
(PMP)®
PMBOK® Guide Discussions  Become a Member Our Leadership
Certified Associate in Project
Online Courses Management (CAPM)® Templates  Student Membership Official PMI Blog 

Events PMI Agile Certified Practitioner (PMI- Blogs  Local Chapters What is Project Management?
ACP)® Stay Connected
Store Volunteering Membership FAQs Who are Project Managers?
Agile Certifications
Explore PMI Careers
Give Feedback

See All Certifications

Report PDUs 
Support
Contact Us

Press and Media

Store Help

Privacy Sitemap Terms of use Purchasing Terms Advertising & Sponsorship © 2023 Project Management Institute, Inc.