Comptroller General

of the United States

United States Government Accountability Office DOCUMENT FOR PUBLIC RELEASE
Washington, DC 20548
The decision issued on the date below was subject to
a GAO Protective Order. This redacted version has
been approved for public release.

Decision
Matter of: IBM-U.S. Federal

File: B-407073.3; B-407073.4; B-407073.5; B-407073.6

Date: June 6, 2013

Jason A. Carey, Esq., Luke W. Meier, Esq., Katherine M. John, Esq., Kevin T.
Barnett, Esq., and John W. Sorrenti, Esq., McKenna Long & Aldridge LLP, for the
protester.
Craig A. Holman, Esq., Kara L. Daniels, Esq., Lauren J. Schlanger, Esq., and
Steffen G. Jacobsen, Esq., Arnold & Porter LLP, for Amazon Web Services, Inc., an
intervenor.
Edwin G. Doster, Esq., Avi M. Baldinger, Esq., and Arthur Passar, Esq., Central
Intelligence Agency, for the agency.
Paul E. Jordan, Esq., and David A. Ashen, Esq., Office of the General Counsel,
GAO, participated in the preparation of the decision.
DIGEST

1. Protest is sustained where agency’s evaluation of protester’s price under one of

the solicitation’s price scenarios was not calculated in such a way as to result in
evaluation on a common basis.

2. Protest is sustained where, during post-selection negotiations, which were

anticipated by the solicitation, the agency materially relaxed a solicitation term
regarding software certification; relaxing this term for only the awardee resulted in a
failure to treat offerors equally.

3. Protester’s contention that the agency’s evaluation of the past performance of a

commercial cloud services provider improperly ignored information from news
reports of service outages is denied where the evaluators were aware of the
reports, but concluded that the reports lacked sufficient detail concerning the
offeror’s adherence to service level agreements to justify downgrading the offeror’s
past performance rating.
DECISION

IBM U.S. Federal (IBM), of Bethesda, Maryland, protests the Central Intelligence
Agency’s (CIA) award of a contract to Amazon Web Services, Inc., of Seattle,
Washington, under request for proposals (RFP) No. 2012-12041000001, for
commercial cloud services (C2S). IBM challenges the evaluation of proposals and
the selection decision.

We sustain the protest in part and deny it in part.

BACKGROUND

The RFP contemplated the award, on a “best value” basis, of a single fixed-price
indefinite-delivery/indefinite-quantity (ID/IQ) contract for commercially-managed
cloud computing services for the Intelligence Community to be provided through
task orders. The contemplated ID/IQ contract was to include a 270-day period to
achieve initial operating capability, a 4-year base ordering period, a 3-year option,
and a 2-year option, with a maximum value of $600 million over the base period.

According to the Federal Cloud Computing Strategy, “cloud computing describes a

broad movement to treat [information technology (IT)] services as a commodity with
the ability to dynamically increase or decrease capacity to match user needs.”
Statement of Objectives at 1. In this regard, cloud computing delivers computing
services via networks with the potential to provide IT services more quickly and at a
lower cost. Cloud computing provides users with on-demand access to a shared
and scalable pool of computing resources with minimal management effort or
service provider interaction. Information Technology Reform: Progress Made but
Future Cloud Computing Efforts Should be Better Planned, GAO-12-756 at 1 (July
2012). By leveraging shared infrastructure and economies of scale, cloud
computing allows users to control the computing services they access, while
sharing the investment in the underlying IT resources among consumers.
Statement of Objectives at 1.

Here, the contractor generally was to provide a copy of its existing public cloud
(modified where necessary) to be installed on government premises and operated
by the provider. Performance is to include both: (1) infrastructure as a service, in
which the provider (contractor) is responsible for networking, storage, servers and
virtualization, and the consumers (government agencies) are responsible for the
operating system, middleware, runtime, data, and applications, System
Requirements Document (SRD) Fig. 2-2; and (2) platform as a service, which
encompasses the same components as infrastructure as a service, but only
applications and data are the responsibility of the consumer.1 Id.

1
The concepts of infrastructure as a service, and platform as a service are also
referenced in GAO’s audit work on cloud computing. Information Technology
Reform: Progress Made but Future Cloud Computing Efforts Should be Better
Planned, supra, at 4.

Page 2 B-407073.3 et al.

In addition, the required intelligence community cloud was to provide for auto-
scaling, SRD § 3.9.1, that is, a mechanism that allows computing infrastructure to
automatically scale up or down the number of computer resources that are being
used to support an application, based on the needs of that application at a specific
time. Contracting Officer’s Statement (COS) at 8. Without this capacity, customers
must anticipate when surge capacity is required and provision it in advance. Id.
Further, of significance for this protest, the solicitation provided that the intelligence
community cloud system (for both infrastructure as a service and platform as a
service) was to permit consumers (here, the intelligence community agencies) to
run their own applications (those not provided by the cloud provider) on the cloud,
and the cloud was to provide “scalable” solutions, with respect to applications,
including those provided by the consumer. See SRD §§ 3.4.11, 3.9.1.

The solicitation provided for an initial mandatory qualification evaluation to verify

that an offeror was an established commercial cloud service provider with an
existing, large-scale public offering. This was to be followed by the evaluation of
proposals under four factors: (1) technical/management, including subfactors for
technical approach (evaluated under a demonstration/oral presentation element and
a written element), service level agreements, and management; (2) past
performance; (3) security, evaluated on a pass/fail basis; and (4) price.

Price was to be evaluated for completeness, reasonableness, and to determine the

offeror’s understanding of the government’s requirements. Evaluation Criteria at 5.
Offerors were required to submit a fixed price for task order 1 for program
management to achieve initial operating capability, and a guaranteed minimum
price for task order 2 for the provision of cloud services for the first year after initial
operating capability. In addition, offerors were to provide a Cloud Services Catalog
Price List containing fixed prices for various services. Also, the solicitation set forth
six representative service-type ordering scenarios; using their proposed catalog
prices, offerors were to calculate the total costs of orders for each scenario, on a
yearly basis, for the base performance period. The total evaluated price was to be
comprised of the sum of the task order 1 pricing and the prices for the six ordering
scenarios for the base period. Price Instructions at 1; Evaluation Criteria at 5.

Five offerors, including IBM and Amazon, submitted proposals by the July 13, 2012,
closing time. Shortly thereafter, two offerors (AT&T Corporation and Microsoft
Corporation) filed protests with our Office challenging various aspects of the
mandatory qualification requirements, and a third offeror withdrew its proposal.
When the agency elected to take corrective action by amending the RFP to remove
the qualification language challenged in the protests, we dismissed both protests as
academic (B-407073, B-473073.2, Aug. 17, 2012).

Based upon the subsequent demonstrations/oral presentations and the evaluation

of proposals by the technical/management evaluation team, past performance

Page 3 B-407073.3 et al.

evaluation team, and price evaluation team, the agency decided to include the
proposals of IBM, Amazon, and a third offeror in the competitive range. After
discussions with the competitive range offerors, the agency requested submission
of final proposal revisions (FPR).

In its evaluation of FPRs, the technical/management evaluation team assigned a

deficiency to IBM’s proposal under the demonstration element (within the technical
approach subfactor) as a result of language in the firm’s FPR that appeared to
indicate that its existing public cloud could not provide auto-scaling for applications
provided by the consumer. The technical/management evaluation team also
assigned a significant weakness under the written element of the technical
approach subfactor for IBM’s failure to make clear in its proposal how IBM would
modify its existing commercial cloud to provide the required auto-scaling capability
for the intelligence community.

In addition, because the offerors appeared to have adopted materially different

interpretations of the scenario requirements, the price evaluation team adjusted
offerors’ prices (both up and down) using each offeror’s proposed catalog prices in
order to “provide a common basis for comparing price.” Report on Final Cost
Proposals at 5. The agency’s adjustment of IBM’s FPR scenario 5 price increased
IBM’s price for this scenario from approximately [deleted] to approximately [deleted].

The final consensus evaluation for IBM and Amazon was as follows:

Amazon IBM
Technical /Management
-- Technical Approach (Demo) Very Good Marginal
-- Technical Approach (Written) Exceptional Very Good
-- Service Level Agreements Very Good Satisfactory
-- Management Approach Satisfactory Very Good
Past Performance (confidence) High Moderate
Security Pass Pass
Proposed Price [deleted] [deleted]
Evaluated Price $148.06 million $93.9 million
Guaranteed Minimum [deleted] [deleted]
Overall Proposal Risk Low High

Source Selection Decision (SSD) at 1.

Based upon the above evaluation results, the source selection authority (SSA)
concluded that Amazon’s proposal represented the best value. In this regard, while
IBM’s proposal offered an evaluated [deleted] price advantage over 5 years, the
SSA concluded that this advantage was offset by Amazon’s superior technical
solution. SSD at 8.

Page 4 B-407073.3 et al.

Prior to contract award, and in accordance with the terms of the RFP, the agency
conducted post-selection negotiations with Amazon to address certain matters in its
proposal. RFP at 47. Upon learning of the ultimate award to Amazon and after
receiving a debriefing, IBM filed this protest.

DISCUSSION

IBM raises numerous challenges to the evaluation of its technical and price
proposals, the conduct of discussions (including the post-selection negotiations with
Amazon), and the evaluation of Amazon’s past performance. Based upon our
review of the record, we find for IBM on its challenges to the agency’s price
evaluation under one of the solicitation’s price scenarios, and the conduct of post-
selection negotiations. For these reasons we sustain the protest. We deny IBM’s
challenges to the agency’s evaluation of Amazon under the past performance
evaluation factor. We have considered all of IBM’s arguments and discuss the most
significant below.

Scenario 5 Evaluated Price

As set forth above, offerors were required to address six notional scenarios.
Scenario 5, for data analytics, was identified as platform as a service (everything
but applications and data are the responsibility of the contractor), and included the
following description:

This scenario centers around providing a hosting environment for

applications which process vast amounts of information in [parallel] on
large clusters (1000s of nodes) of commodity hardware in a reliable,
fault-tolerant manner (MapReduce). The solution to this scenario
should automatically provision clusters of compute for the
segmentation and [parallel] processing of input datasets via the
MapReduce framework (3.4.1) where the vendor is responsible for the
management of the OS [operating system] and MapReduce
implementation. Assume a cluster large enough to process 100TB
[terabytes] of raw input data. Assume input data set was loaded from
available object-based storage that realizes 6 reads/second and
2 writes/second. Assume 100% duty cycle on all virtual machines
associated with this scenario.[ 2] (Total storage for each order shall be
100 TB object. Read/writes and IOPS [input/output operations per
second] should be calculated per order).

2
Duty cycle refers to the percentage of time a computer spends in an active state
as a fraction of the total duration. IBM First Supplemental Protest at 40. If the
duration is 1 year, a 100% duty cycle would cover 8,760 hours of time. COS at 33.

Page 5 B-407073.3 et al.

RFP, amend. 4, Scenario 5 Description (emphasis added).

Pointing to the solicitation reference to a “100% duty cycle” and the pricing
instruction’s request for yearly prices, the agency explains, in its report and at the
hearing our Office conducted in this matter, that it was requesting under scenario 5
the prices for simultaneously running a large number of orders, each consisting of
100 TB of data, repeatedly throughout the year. COS at 28-29; Hearing Transcript
(Tr.) 269-71. Because IBM’s FPR price was based on a single run processing
100 TB of data, multiplied by the specified number of orders for the period (30 for
the first year), the agency used IBM’s proposed catalog pricing and the services
identified in its FPR for this scenario, to adjust the firm’s price to represent 1 year of
continual processing for each order.

IBM asserts that the agency’s adjustment of its scenario 5 price was unreasonable.
In this regard, IBM maintains that the agency imposed an unstated requirement that
each 100 TB data analytics unit provided under this scenario was to be continually
repeated throughout the entire year. In contrast, under the protester’s interpretation
(as reflected in its FPR), it was only required to propose a solution capable of
processing a single 100 TB data run with no specified duration. IBM First
Supplemental Protest at 40. IBM explains that it therefore proposed an “optimal”
price for a level of services (e.g., computer capacity) with a processing time of
approximately [deleted], which represented a single run of 100 TB of data.
Tr. at 410.

As an initial matter, we find untimely IBM’s challenge to the agency’s interpretation

that scenario 5 called for repeated 100 TB data runs throughout the year, rather
than a single run under each order. In this regard, prior to submitting its initial
proposal, IBM asked two questions seeking additional information about the
scenario’s technical parameters. Specifically, IBM first asked:

Are orders the number of new images of that scenario type that are
loaded to the image store/catalog in the year or are they the number
of instantiations/runs of that scenario type in a year?

Question/Answer No. 49. The agency responded that “[a]s outlined in the scenario,
the servers should be treated as operating on 100% duty cycle and should be
priced out as simultaneous orders.” Id.

IBM’s second question was more detailed, requesting detailed information

concerning the scenario requirements. Among other things, IBM’s second question
asked if concurrent data analytics users would be running jobs, what would be the
anticipated average number of runs of each scenario type daily, monthly, etc., and
how many analytic jobs were expected to be concurrently executing.
Question/Answer No. 50. In response, the agency repeated its answer to Question
No. 49, and added:

Page 6 B-407073.3 et al.

 The contractor should propose commercial best practices derived
from their commercially available solution(s) to provide data analytics
via the MapReduce software framework to concurrent users from
multiple organizations.

Id.

Without further inquiry, IBM initially priced scenario 5 at approximately [deleted],

based on the specified number of orders, with 12 months of availability. Thus,
IBM’s initial approach, consistent with the manner in which all other offerors in the
competitive range prepared their pricing, appeared to be based upon continual
100 TB data runs throughout the year. COS at 32-33; IBM Initial Proposal at VI-17
(proposal of services at 12 months each); Tr. at 428-29 (“7/24/365” capacity
approach). However, in its FPR, IBM (but not the other offerors) reduced its
scenario 5 price to approximately [deleted], based on a single run for each of the
specified number of orders.

Under our Bid Protest Regulations, a solicitation defect apparent on the face of the
solicitation must be protested prior to the time set for receipt of initial proposals or
quotations, when it is most practicable to take effective action against such defects.
4 C.F.R. § 21.2(a)(1) (2013). Furthermore, an offeror who chooses to compete
under a patently ambiguous solicitation does so at its own peril, and cannot later
complain when the agency proceeds in a way inconsistent with one of the possible
interpretations. Wackenhut Servs., Inc., B-276012.2, Sept. 1, 1998, 98-2 CPD ¶ 75
at 5; CardioMetrix, B-274585, Nov. 18, 1996, 96-2 CPD ¶ 190 at 3; Watchdog, Inc.,
B-258671, Feb. 13, 1995, 95-1 CPD ¶ 69 at 5.

Here, by its own actions, IBM evidenced its recognition that the scenario 5
instructions were ambiguous as to the frequency of the expected 100 TB data runs.
IBM requested clarification of the requirements in this regard and then, not having
received meaningful clarification, first adopted one interpretation (that is, continual
runs) in its initial proposal and then a different interpretation (a single run per order)
in its FPR. Having chosen to compete despite its recognition of the patently
ambiguous nature of the solicitation in this area, IBM cannot now complain when
the agency proceeds in a manner inconsistent with one of the possible
interpretations and adjusts IBM’s price to match the government’s (and Amazon’s
apparent) interpretation of the requirement. See, e.g., Wackenhut Servs., Inc.,
supra.

That said, while we view as untimely IBM’s challenge to the agency’s interpretation
that continual runs (100% duty cycle for 1 year) were required under scenario 5, its
challenge to the agency’s calculation of an evaluated price for scenario 5 is not
untimely.

Page 7 B-407073.3 et al.

In this regard, our Office will review the reasonableness or consistency of the
evaluation method the agency employs. See Federal Computer Int’l Corp.,
B-276885, July 29, 1997, 97-2 CPD ¶ 35 at 3. In meeting the requirement to
consider cost or price to the government in evaluating competitive proposals,
41 U.S.C. § 3306(c)(1)(B) (2011), an agency’s chosen method of evaluation must
include some reasonable, common basis for evaluating or comparing the relative
costs of proposals. See Aalco Forwarding, Inc., et al., B-277241.15, Mar. 11, 1998,
98-1 CPD ¶ 87 at 11. Ordinarily, normalization--which is the term used by the
parties here--involves the measurement of offerors’ costs against the same baseline
where there is no logical basis for differences in approach or where there is
insufficient information provided with the proposals, leading to the establishment of
common estimates by the agency. See Bendix Field Eng’g Corp., B-246236,
Feb. 25, 1992, 92-1 CPD ¶ 227 at 17.

Here, the record indicates that the agency lacked sufficient information to ensure
that proposals were evaluated on a common basis with respect to scenario 5. In
this regard, the agency attempted to ensure that offerors were evaluated on a
common basis with regard to scenario 5 by scaling IBM’s single-run price upwards
to equal one year of continual processing for each order. Generally, this resulted in
assuming that each IBM 100 TB data run, indicated by IBM to take [deleted], would
be repeated approximately [deleted] times in a year. Tr. at 361, 375.

However, while Amazon’s scenario 5 price indicated that it, like every other offeror
besides IBM, assumed repeated 100 TB data runs throughout the year, an agency
adviser to the price evaluation team (and author of scenario 5) testified that based
on the information in the proposal, the agency did not know how long a single
100 TB data run would take using Amazon’s proposed solution. Tr. at 314, 378.
Specifically, Amazon’s proposal made various assumptions about running 100 TB
data sets at “100% duty cycle” and “leveraging [its] [deleted],” and “priced [its]
[deleted]” instance type for 100% of the time, but it did not propose a specific time
for a single 100 TB data run. Amazon Proposal at VI-32-33. Thus, there was no
way from the information available concerning Amazon’s solution to ascertain how
many 100 TB data runs were included in Amazon’s scenario 5 pricing.

Further, the agency recognized that solutions (and prices) could vary based on the
makeup of virtual machines--the amount of locally available storage, the amount of
memory, the amount of virtual compute cores proposed--and how fast a unit
processes 100 TB of data, all of which would have an effect on performance,
including how many times a solution could process that amount of information in a
given period. Tr. at 344, 347. In this regard, while Amazon’s proposed scenario 5
solution included [deleted] compute units or [deleted] virtual cores (i.e., divisions
within the virtual computer available for separately processing information without
interfering with one another), IBM’s solution included [deleted] virtual cores;
[deleted]. Tr. at 386-88.

Page 8 B-407073.3 et al.

In sum, given the agency’s uncertainty regarding just what performance (e.g.,
number of 100 TB data runs) was included in each evaluated price, there is no
basis for concluding that Amazon was evaluated for scenario 5 using the same or
otherwise comparable level of performance as included in IBM’s adjusted price.
Thus, we find the price evaluation to be unreasonable and we sustain the protest on
this basis.

Post-Selection Negotiations

The RFP included the following provision regarding software certification:

The Contractor certifies that it will undertake to ensure that any

software to be provided . . . under this contract will be provided . . .
free from computer virus, which could damage, destroy, or maliciously
alter software, firmware, or hardware, or which could reveal to
unauthorized persons any data or other information accessed through
or processed by the software.

RFP Commercial Clause § 152.204-706(a). In its post-selection negotiations with

Amazon, Amazon proposed, and the agency accepted, modifying the above
solicitation clause as follows: “The Sponsor agrees that ‘only software developed
and provided by [Amazon] would be subject to this requirement.’” Memorandum of
Understanding, Feb. 12, 2013, at 3.

IBM asserts that in agreeing to this modification, the agency relaxed a material
requirement of the solicitation. The agency responds that post-selection
negotiations were contemplated by the RFP, RFP at 47 (“The Government intends
to select for final negotiations a contractor(s) resulting from this solicitation whose
proposal represents the best value . . . .”), and that this modification does not
represent a material change to the RFP’s requirements or Amazon’s obligations.
We disagree.

It is a fundamental principle of government procurement that competition must be

conducted on an equal basis; that is, offerors must be treated equally and provided
with a common basis for the preparation of their proposals. Systems Mgmt., Inc.;
Qualimetrics, Inc., B-287032.3, B-287032.4, Apr. 16, 2001, 2001 CPD ¶ 85 at 8.
When, either before or after receipt of proposals, the government changes or
relaxes its requirements, it must issue an amendment to notify all offerors of the

Page 9 B-407073.3 et al.

changed requirements and give them an opportunity to respond. Diebold, Inc.,
B-404823, June 2, 2011, 2011 CPD ¶ 117 at 4; Systems Mgmt., Inc.; Qualimetrics,
Inc., supra; see Cardkey Sys., B-220660, Feb. 11, 1986, 86-1 CPD ¶ 154 at 2 (If it
becomes apparent that the contract being negotiated differs significantly from the
requirements stated in the RFP, the contracting agency must amend the RFP or, at
the least, advise offerors of the change during discussions and seek new offers.)
We will sustain a protest where an agency, without issuing a written amendment,
materially alters the solicitation’s requirements to the protester’s prejudice. See
Systems Mgmt., Inc.; Qualimetrics, Inc., supra.

Here, the record indicates that the modification of the RFP clause to the negotiated
language materially reduced Amazon’s obligations from those imposed in the RFP.
In this regard, the RFP clause required the offeror to certify its undertaking to
ensure that “any software” provided will be virus free, while the modified language
covered “only software developed and provided” by Amazon. This is significant
because Amazon’s proposal contemplated the provision of third party and open-
source software that is not developed by Amazon and thus would not fall within its
modified certification. See, e.g., Amazon Proposal at 1-4,1-9 (“rich ecosystem of
. . . third-party security tools”), and at 1-13, 1-23, 1-26, 2-38 (proposed use of
“MySQL” and “Red Hat Enterprise Linux,” an open-source database and software
product, respectively).

During the hearing before our Office, the agency admitted that, at the time of the
post-selection negotiations, it did not consider the impact of the modified language;
it understood that Amazon was certifying all software and did not realize that the
change might be a restriction on the certification. Tr. at 486-88. Indeed, the
contracting officer testified that had the agency focused more closely on the
modification, it would have taken exception to the revision. Tr. at 502-03. Thus, the
hearing testimony suggested that the new language represented a material change
to the terms of the underlying competition.

Both at the hearing and in briefs filed after the hearing, however, the agency asserts
that this change was not significant because Amazon is still required to comply with
other provisions of its contract. For example, the agency points out that SRD
§ 3.13.2 required C2S infrastructure and services to be “assessed and authorized in
accordance with Intelligence Community Directive (ICD) 503 (reference 3),” which
provides for the agency to incorporate standards, policies, and guidelines issued by
the National Institute of Standards and Technology and the Committee on National
Security Systems (CNSS). Agency Post Hearing Comments at 20. One aspect of
these standards includes “malicious code protection,” and Amazon’s detailed
proposal to meet these requirements was incorporated into the contract.
Tr. at 470-72; Amazon System Security Plan, Appendix B-91--B-93.

We see no basis in the contemporaneous record to support a conclusion that the

agency made its decision to allow the modification based on these other provisions

Page 10 B-407073.3 et al.

of the solicitation, or on Amazon’s agreement to them.3 In this regard, while we
consider the entire record in resolving a protest, including statements and
arguments in response to a protest, in determining whether an agency’s selection
decision is supportable, we accord greater weight to contemporaneous evaluation
and source selection materials than to new judgments made in response to protest
contentions. Boeing Sikorsky Aircraft Support, B-277263.2, B-277263.3, Sept. 29,
1997, 97-2 CPD ¶ 91 at 15. We accord lesser weight to post hoc arguments or
analyses because we are concerned that judgments made in the heat of an
adversarial process may not represent the fair and considered judgment of the
agency, which is a prerequisite of a rational evaluation and source selection
process. Id. Here, given the contracting officer’s testimony that, had the agency
focused on the modification requested by Amazon, it would have taken exception,
we find no basis for accepting the agency’s current assertion that the modification
was not material.

In any case, the agency’s current position does not show that the modification was
not material. In this regard, the agency now contends that the only aspects not
covered by Amazon’s agreement to the certification requirements were (1) the
certification itself, which the agency believes would be satisfied by Amazon signing
its contract; and (2) the requirement to immediately notify the contracting officer of
an issue with malicious code. Agency Post Hearing Comments at 20. However,
regardless of the agency’s position with respect to the certification, it is unclear why
elimination of Amazon’s obligation to immediately notify the contracting officer about
malicious code would not represent a change in a material requirement.4

3
The agency also generally claims that the SSA was briefed on the final negotiation
items and felt this was a minor issue. Tr. at 469; SSD at 9. The contracting officer
was unaware of the impact of this particular provision and there is nothing in the
record to indicate that he provided a relevant assessment of that impact in the
briefing to the SSA or that the SSA exercised otherwise considered judgment in this
regard.
4
Our conclusion is not changed by Amazon’s assertions that IBM also sought
numerous proposed changes to provisions in the RFP, including a proposal that it
would not provide [deleted] with respect to [deleted]. IBM FPR at 5-8; see
Intelligent Env’ts, B-256170.2, Nov. 28, 1994, 94-2 CPD ¶ 210 at 4 (protest denied
where neither protester nor awardee complied with specific requirement). While
offerors were free to ask for changes, waiving a material term of the solicitation for
only one of them, after the selection decision was made, was improper. We also
disagree with Amazon that IBM cannot claim to be prejudiced by this change. IBM
argues that this modification materially reduces the risk to the contractor. IBM
Supplemental Comments, May 6, 22013, at 8. In our view, IBM’s assertion that the
level of risk to the contractor was reduced by this modification is sufficient to
establish prejudice.

Page 11 B-407073.3 et al.

In sum, we find that the agency, without issuing a written amendment, materially
relaxed the solicitation’s requirements for Amazon without affording the other
offerors an opportunity to propose to the modified requirements. Accordingly, the
protest is sustained on this basis as well.

Auto-Scaling Evaluation

During the agency’s final evaluation, the evaluators assigned IBM’s proposal a
deficiency under the demonstration factor because they found that IBM’s FPR
introduced new language that cast doubt on whether the firm’s existing public cloud
provided auto-scaling for all applications, both IBM-supplied and agency consumer-
supplied. COS at 9. While the evaluators found that IBM’s proposed C2S cloud for
the intelligence community included auto-scaling capability for all applications, they
also assigned a significant weakness because there was a lack of detail on how this
additional capability (relative to the existing public cloud) would be accomplished.
Id. at 13. IBM asserts that the agency’s evaluation was irrational and that its
proposal made clear that its public cloud provided auto-scaling capability for all
applications whether supplied by IBM or agency consumers.

In considering protests challenging an agency’s evaluation of proposals, we will not

reevaluate proposals; rather, we will examine the record to determine whether the
agency’s evaluation conclusions were reasonable and consistent with the terms of
the solicitation and applicable procurement laws and regulations. Gonzales-Stoller
Remediation Servs., LLC, B-406183.2 et al., Mar. 2, 2012, 2012 CPD ¶ 134 at 5. A
protester’s disagreement with a procuring agency’s judgment is insufficient to
establish that the agency acted unreasonably. Sig Sauer, Inc., B-402339.3, July 23,
2010, 2010 CPD ¶ 184 at 6.

Here, the proposal instructions required each offeror to “clearly demonstrate its
understanding of the technical requirements and suitability of the proposed
technical approaches that will be employed.” Evaluation Criteria at 2. Moreover, it
is an offeror’s obligation to submit an adequately written proposal for the agency to
evaluate. See Independence Constr., Inc., B-292052, May 19, 2003, 2003 CPD
¶ 105 at 5.

Our review of the record shows that the RFP clearly required offerors to
demonstrate the ability to auto-scale computer resources both in their existing
public cloud and in the C2S cloud solution. Specifically, the cloud was to provide
“dynamic workload management . . . across the cloud infrastructure,” SRD § 3.4.11,
and “shall provide application services with . . . automated scaling.” SRD § 3.9.1.
While IBM’s initial presentation demonstrated its auto-scale capability for
applications from its existing public cloud structure, in its FPR, IBM revised its
written proposal in support of the demonstration so as to indicate that its ability to
auto-scale in its existing public cloud structure was limited to applications provided
by IBM. In this regard, in referring to its capability to auto-scale, IBM’s FPR drew a

Page 12 B-407073.3 et al.

distinction between [deleted]. IBM FPR at 1-25.5 This distinction was continued in
a figure from its FPR indicating that, [deleted].6 Id. at 1-26. Thus, IBM’s FPR
suggested that the ability to auto-scale in its existing public cloud structure was
limited and did not apply to all services.

As explained by the agency, it had “grave” concerns about IBM’s auto-scaling

capability when it considered these limiting references in conjunction with a
subsequent statement in IBM’s FPR that an application was to be selected based
on “catalog choices.” IBM FPR at 1-28; Tr. at 50-51. According to the agency, it
interpreted this and other uses of “catalog” in IBM’s proposal to refer to the C2S
service catalog, a listing of IBM’s fixed prices for cloud services. Supplemental
COS at 3. While agency consumers could add their own applications to an overall
“library” for their use (SRD § 3.4.13), the IBM service catalog could not be altered
by agency consumers. Id. Thus, given the indications in IBM’s FPR that auto-
scaling in its existing public cloud structure was limited to applications from IBM’s
service catalog, and that consumer applications could not be added to that catalog
by the consumer, the evaluators could not conclude that IBM’s existing public cloud
structure, as described in its FPR, provided a mechanism through which an
application furnished by an agency consumer could be auto-scaled. Final Technical
Evaluation Report at 8.

In response to the agency’s arguments, the protester identifies a number of places

in its FPR where it used the word catalog in a more generic sense, thus indicating
that its reference to selecting applications from the “catalog” was not intended to
mean the service catalog provided by IBM. For example, IBM’s FPR stated that the
“[intelligence community] can build custom applications for inclusion in the image
library or catalog.” FPR at 2-7, 2-8. In addition, at a later stage of the
demonstration, where IBM showed its ability to add consumer applications to a
library, the IBM presenter stated that “that image now exists in the catalogue.”
Tr. at 113. IBM also notes that even the RFP uses the term catalog other than in
the context of the services catalog. Specifically, it refers to the requirement that an
offeror provide an image library, a repository for virtual images and VM templates
“to be managed and selected from the catalog.” SRD § 3.4.13.

Our review of the record indicates that these various references suggest that IBM
may in fact have auto-scaling capability for consumer applications. However, while
the cited passages in IBM’s FPR established, e.g., that consumer applications can
be added to the “library,” and that “applications” can be auto-scaled in IBM’s

5
In providing both platform and infrastructure as services, an “instance” refers to
the supply of a virtual machine or server for use by the consumer. Tr. at 45-46.
6
Load balancing refers to the distribution of computer workload across multiple
servers, while auto-scaling refers to changing the capacity of a particular server.

Page 13 B-407073.3 et al.

existing public cloud structure, there is no place in the FPR where IBM clearly and
affirmatively stated that its existing public cloud structure can auto-scale all
applications, whether provided by IBM or the consumer. Tr. at 70, 220. Likewise,
there is no clear statement in the FPR defining what the applications catalog is as
distinguished from the IBM-supplied service catalog.

We recognize that IBM’s proposed cloud for the intelligence community--i.e., the
contract at issue here--promised the capability to auto-scale all applications.
However, the above deficiency was assigned for failure to demonstrate the
existence of this capability in IBM’s existing public cloud, a different element of the
evaluation. Further, since we find the agency reasonably determined that IBM
failed to clearly establish the capability of its existing public cloud to auto-scale all
applications, there is no basis to find unreasonable the agency’s assignment of a
significant weakness, under a different evaluation element (the written element), as
a result of IBM’s failure to explain how it would add the capability to auto-scale all
applications in the cloud it proposes to provide for the intelligence community.

We see no basis to question the reasonableness of the agency’s concerns

(expressed as a deficiency and a significant weakness under the technical
approach subfactor) that IBM’s proposal failed to adequately explain its auto-scaling
capabilities. Accordingly, this aspect of the protest is denied.

Past Performance Evaluation

IBM also argues that the past performance evaluation of Amazon’s proposal was
flawed because the agency failed to take into account news reports about
five service outages experienced by some of Amazon’s public cloud clients between
April and December 2012, while the agency was conducting this procurement.7
IBM First Supplemental Protest at 57. IBM contends that this information was
relevant to the past performance evaluation, noting that the second most important
element under the past performance factor was whether an offeror’s “performance
history demonstrates success in meeting management, delivery, and performance
requirements within the terms of a Service Level Agreement [SLA].” Evaluation
Criteria at 3-4. IBM argues that had the agency considered this information,
Amazon’s past performance rating would have been lower than high confidence.

Our Office has recognized that in certain limited circumstances an agency

evaluating an offeror’s past performance has an obligation (as opposed to the
discretion) to consider outside information bearing on an offeror’s past performance.
New Orleans Support Servs. LLC, B-404914, June 21, 2011 CPD ¶ 146 at 5.

7
These outages were reported in various media outlets, including, for example,
The Washington Post, The New York Times, The Wall Street Journal, and Cable
News Network. IBM First Supplemental Protest, Exhs. D through P.

Page 14 B-407073.3 et al.

Where we have charged an agency with responsibility for considering such outside
information, the record has demonstrated that the information in question was “too
close at hand” to require offerors to shoulder the inequities that spring from an
agency’s failure to obtain, and consider, the information. Id.; see e.g., International
Bus. Sys., Inc., B-275554, Mar. 3, 1997, 97-1 CPD ¶ 114 at 5; G. Marine Diesel,
B-232619.3, Aug. 3, 1989, 89-2 CPD 101 at 4-6 (contracting officer who was
personally aware of the awardee’s continuing difficulties in performing a contract for
services related to the subject solicitation, and who considered those difficulties in
deciding not to exercise the remaining options, erred in not considering the
difficulties when evaluating proposals).

The record here shows that the contracting officer, the SSA, and other members of
the evaluation team, were aware of the news reports regarding the service outages,
Addendum to Supplemental Agency Report at 1, but concluded that the reports and
their knowledge of the circumstances surrounding the outages were not sufficiently
detailed to take the outages into account in the past performance evaluation. Id.;
Contracting Officer’s Declaration, Apr. 25, 2013; SSA Declaration, Apr. 26, 2013.
For example, as noted by the SSA, none of the cited news reports addressed
Amazon’s success in meeting the terms of its SLAs with the affected customers.
SSA Declaration, Apr. 26, 2013.

In addition, the record shows that one of Amazon’s past performance references
acknowledged outages in Amazon’s system. Specifically, the National Aeronautics
and Space Administration, Jet Propulsion Laboratory (JPL), rated Amazon as very
good for meeting SLAs, despite reported outages, commenting:

[Amazon] has experienced some outages, especially in their Virginia

region. However, this has had no impact on JPL, and as a percentage
of the overall availability, these outages are miniscule. All cloud
vendors have outages. All private clouds and private data centers
have outages. The difference is that we hear about them when they
happen in a public cloud, such as [Amazon]. JPL operates mission
critical services in the [Amazon] cloud and we have had 100% uptime
since inception. To achieve this, we implement failover and elastic
load balancing but it’s simple and inexpensive and very much worth it.

JPL Past Performance/Questionnaire at 3.

In conclusion, we see no basis to find that the agency improperly evaluated

Amazon’s proposal under the past performance factor.

Evaluation of Guaranteed Minimum Prices

IBM asserts that the agency improperly considered the effect of its guaranteed
minimum price on its total evaluated price. In this regard, IBM relies on the RFP

Page 15 B-407073.3 et al.

language that the guaranteed minimum would not be used as part of the total
evaluated price. Evaluation Criteria at 5. IBM’s assertion is without merit.

The guaranteed minimum price feature of this solicitation was intended to allow the
offerors to recover a portion of the upfront infrastructure expenditures and to reduce
their cost risk. Source Selection Evaluation Board (SSEB) Recommendation at 19.
The RFP specifically provided that the guaranteed minimum proposed by each
offeror would be evaluated for completeness and reasonableness and would “be
used within the Government’s overall risk rating involved with this acquisition.”
Evaluation Criteria at 5. The SSEB, in its discussion of the offerors’ guaranteed
minimum prices, addressed the risk involved with IBM’s minimum by noting that
[deleted]. SSEB Report at 20. The SSEB also found “serious cost risk” associated
with IBM’s proposed contract clause which indicated that the contractor would
[deleted]. Id. The SSEB observed that if the guaranteed minimum was “considered
as part of the likely cost,” it would reduce the price difference between IBM and
Amazon. Id. The SSA made similar observations in her source selection decision.
SSD at 9.

In our view, the evaluation in this regard was not inconsistent with the RFP’s
evaluation criteria. The record shows that IBM’s evaluated price was not increased
by the amount of its guaranteed minimum. See, e.g., SSD at 8. Rather, the
SSEB’s evaluation and the SSA’s tradeoff analysis simply considered the risk
involved in IBM’s guaranteed minimum and future position on negotiations. While
IBM objects to consideration of the magnitude of its guaranteed minimum, clearly
the magnitude of the guarantee was a factor in the extent of the resulting risk, which
under the RFP was to be evaluated.

RECOMMENDATION

We sustain the protest because the agency’s adjustment of scenario prices was
unreasonable in that it did not result in evaluation on a common basis, and because
the agency materially relaxed a solicitation term for the awardee during post-
selection negotiations. We recommend that the agency reopen the competition and
amend the RFP as necessary to ensure that proposals are prepared and evaluated
on a common basis, consistent with the issues discussed in this decision. We
further recommend that the agency conduct discussions with offerors, obtain and
evaluate revised proposals, and make a new source selection decision. Finally, we
recommend that the protester be reimbursed its costs of filing and pursuing the
protest, including reasonable attorneys’ fees. 4 C.F.R. § 21.8(d)(1).

Page 16 B-407073.3 et al.

The protester’s certified claim for costs, detailing the time expended and costs
incurred, must be submitted to the agency within 60 days after receipt of this
decision. 4 C.F.R. § 21.8(f)(1).

The protest is sustained.

Susan A. Poling
General Counsel

Page 17 B-407073.3 et al.