DevOps with Quality

Achieving the desired quality at every stage of the DevOps lifecycle.

DevOps: increasing Speed and Quality Not overcoming these challenges means that organizations
either have to accept higher risks of IT product release failures
DevOps is being embraced by the IT industry as the answer to
– with potentially serious consequences for business
fulfill the continuous demand from the Business to deliver new
operations and corporate image, or have to accept lower
and improved IT solutions to end-users and customers at ever
speed in adapting new technologies – with potential risk of
increasing speed.
losing to the competition.
The core idea behind DevOps is to remove the boundaries
In this point of view document we will offer some concrete
between the traditional silos of Business, IT Development, QA
solutions that will enable the achievement of the benefits of the
and IT Operations and to extend the agile and lean-principles
DevOps philosophy while minimizing the risk to brand image
from software development to the software deployment
and user experience.
phase. This would ultimately lead to an IT process with a
continuous stream of IT updates to the end-user. The The characteristics of DevOps
proliferation of DevOps is driven by time to market with
Business Agility and business speed to market is more than
business metrics, such as Return on Revenue and Return on
ever linked to agility of IT provisioning:
Investment (ROR/ROI) for new development, and throughput
efficiencies for legacy systems. In our view, to begin to realize • Monolithic legacy systems and lengthy release cycles often
the benefits promised by the DevOps philosophy, automation impede an organizations ability to quickly adapt to new
of the quality activities is not only required but it is the core business processes, because it is still common for releases
enabler of increasing throughput and velocity. Without to be done quarterly or even only twice a year
maximizing automation, the speed at which a team can deploy • Newer and more nimble solutions provide faster ROI
features is limited by the speed at which the quality activities through operational improvements
can be successfully completed. In other words, testing • Agile methods deliver usable product to market in smaller,
faster increments and reduce risk by identifying issues
activities, done traditionally will become the constraining factor
earlier in the lifecycle
or the alternative is testing will not be done adequately,
• Lean processes and advanced automation tooling
therefore putting your organizations reputation at risk.The 2015
empower organizations to adapt to changes at a previously
World Quality Report reveals some significant data points to
unachievable rate
support this view:
• Future advancements in testing will include the
• 61% of organizations rate, time to market as a very incorporation of artificial intelligence and predictive analytics
important part of their corporate strategy that will allow even further improvements in automation of
• Corporate image is the number 1 executive concern when it delivery.
comes to quality, demanding protection from software By adopting DevOps an organization can dramatically improve
glitches that hit the news the value delivered by its business. DevOps is based on two
• Customer Experience or Customer Value is a key objective, principles
and this is determined by speed, fit for purpose solutions
and ease of use. • Collaboration and industrialization leveraging highly
automated approaches to deploy solutions that evolve as
However, the same World Quality Report survey also shows fast as the business needs it.
that achieving the appropriate level of quality validation within • A team centric ethos that tears down traditional silos to
these ever shorter cycles is a serious challenge for most tightly integrate business, development and operations to
organizations: drive agility and service delivery excellence across the entire
lifecycle.
• 49% of organizations complain that still largely manual
testing phases are a bottleneck in speeding up the Implementation of DevOps requires a changed focus on three
development cycle time aspects
• 33% of organizations have difficulty to determine the right • Cultural: Business/Development/QA and operations must
coverage of quality validation checks in Agile and work in a fully integrated fashion in the software delivery
DevOps processes process; Change and adaptability are the norms.
• 31% of organizations claim that test environments and test • Process: an agile based approach with continuous
data sets are not flexible enough to quickly test development, continuous build, continuous integration,
micro-services. continuous testing, continuous deployment and
• Proliferation of 3rd party services integrated into continuous monitoring
applications increase the risk that changes in these services • Tools: implementation of a flexible tool environment that
cause unexpected failures allows maximum automation In order to achieve the desired
pace of delivery while protecting the brand and
user experience.

DevOps with Quality 2

 Testing the way we see it

DevOps is a revolution for all disciplines: Business, 2. Continuous Development based on TDD (Test
Development, Operations and Quality Assurance. All Driven Development ) and BDD (Behaviour Driven
disciplines come together and will integrate extensively to Development). These methods enable relatively quickly a
make DevOps work. Each has a role to play. common shared understanding of the expected
operations and functionality of the application among all
We believe that four key aspects must be made part of your
team members (users, developers, testers and
DevOps process, teams and infrastructure in order to achieve
operations). By extending these descriptions with priorities
DevOps with the appropriate and fit-for-purpose level of
and potential risk identifications, they can be used as input
quality:
to automatically generate test cases.
1. Implementing an integrated DevOps Quality Approach 3. Quality is everyone’s responsibility. At each step in
2. Assigning the role of DevOps Quality Engineer to the DevOps cycle team members execute quality checks:
DevOps teams At continuous build the focus is to validate whether this
3. Providing a coherent yet easy adaptable DevOps Quality build could break operations; at continuous development
Automation Framework the focus is to validate functionality and usability. At
continuous integration the focus is to validate
4. Providing a one-click DevOps Test Environment and Test
performance, security and usability.
Data solution to teams
4. Clear understanding of priorities and potential risks
Implement a DevOps Quality Approach to customer value and business performance. The
quality ambition of DevOps deployments must be targeted
We believe an integrated DevOps Quality Approach is based
at the level “good-enough”. Pre-requisite is an agreed and
on seven key principles
commonly shared vision on risks that are acceptable and
1. Continuous Testing and Continuous Quality risks which cannot be taken with a deployment. The
Monitoring are completely connected to the continuous agreed priorities and identified risk areas are the common
development, continuous build and continuous starting point for all team members. A priority/risk charter
deployment process. per application which is automatically updated based on

Figure 1. DevOps Quality Approach

Sprint

Business Plan Requirements Build & Release

Priorities Development Testing Monitoring
& Release Plan Backlog Deploy Control

a. Zero Touch Continuous Testing: A fully automated customer and end-user feedback is used to enable this.
process for testing is in place, covering all steps of the Predictive analysis from combined logging of historical
test process: from test strategy, test planning, test case project and deployment data-facts with actual quality
generation, test environment set up, test data monitoring scores is used to automatically identify the risk
provisioning, to test execution and covering functional as and focus areas of test and the selection of the
well as non-functional testing aspects (especially appropriate test cases.
performance and security). Quality Analysis tools are 5. Quality Indicators per application. Per application
used to switch on and switch off validation checks with a there is a defined set of measurable quality indicators with
push on the button. Newly defined/created quality pre-defined targets. The standard quality indicators cover
validation checks are automatically absorbed in the full four aspects: fit for purpose, ease of use, performance
available set of quality checks. The test and quality and security. This standard set can be expanded
checks are focused at: essential functional operations, depending on the specific objectives of an application.
performance, security and ease of use. The operational scores of the quality indicators are
b. Continuous Quality Monitoring; The actual quality monitored continuously based on production incident
level of applications is continuously monitored based on reports and end-user feedbacks. Meeting the quality KPI’s
instantaneous visibility of production incidents, user is a shared responsibility for all team members.
reports.

3
 Figure 2. DevOps Quality Metrics Model

Sprint / Engagement Effort Variance

Test Execution Productivity
Iteration Level Metrics
Schedule Variance
Velocity
Commitment Reliability

Defect leakage Number of Releases / Year

Release Business KPI
Severity 1 and 2 defects Reduction in Defects and Support Calls

Features delivered per release Release Predictability Measure

6. Flexible quality coverage decision; For each DevOps (measured through the metrics: Defect leakage, Number
deployment cycle the appropriate quality validation checks of high Severity Defects, Number of features delivered)
and test cases are defined and/or selected based on Engagement Level (Effort variance, schedule variance and
objective rules and with help of predictive analysis tools. commitment Reliability) and Business Level, which is
7. Implement metrics to manage QA efficiency: A small measured by Number of releases per year, Reduction in
set of metrics is used to manage the efficiency of Quality Defects and Production calls, Release Predictability).
activities. These metrics cover four aspects: Sprint/ These metrics are continuously reported in a
Iteration performance (measured through Test Velocity real-time dashboard.
and Test Execution Productivity); Release Quality

Figure 3. Measure of Success

Success Levers Basic Intermediate Advanced

Lifecycle Automation Functional, non functional, Data , UAT 50% 70+%

Continuous Test Build, integrate and deliver 70% 100%

Lean and Agile adoption Kanban, Scrum,Safe,XP,SaFe 40% 70+%

Test virtualization Services and Environment automation 30% 40+%

Test optimization Risk based testing and OA leverage 40% 50%

Standardization Model based work benches 40% 60%

Continuous monitoring Availability & sentiment analysis 30% 60+%

QE leverage TDD, BDD, Unit automation 70% 100%

DevOps with Quality 4

 Testing the way we see it

The key success factors to achieve DevOps with Quality are: 1. Provide QA checkpoints/guidelines for all processes and
all team members
• Lifecycle test automation
2. Provide a common backbone QA Automation platform
• Continuous Testing
3. Provide flexible QA Environments with service virtualization
• Lean and Agile process adoption
and cloud based environments
• Test Virtualization
4. Run end-to-end QA validation checks per application
• Test Optimization and Standardization
5. Validate correct Quality Coverage across the
• Continuous Monitoring
DevOps cycle
• Acknowledge the role of Quality Engineer
6. Validate and Monitor the overall Quality level of
Assign the role of DevOps Quality the applications
Engineer to DevOps teams Empowerment of teams is an important aspect of DevOps.
DevOps with Quality requires the presence of multiple skills Efficiency in the empowerment of the QaA activities requires a
within the teams: base set of re-usable guidelines, frameworks and best
Business and domain skills: DevOps means anything that practices. To manage and provide this a DevOps QMO (
eliminates bottle-necks between concept to deploy. Without Quality Management Office) could be established. This QMO
understanding the unique needs of domains, it is impossible is not intended to act as a new “silo” with execution
to design an appropriate quality strategy that drives validation responsibility for quality validation activities, as its core
activities upstream. function is to provide the QA validation standards and core
tool platform and frameworks across teams.
Architecture skills: Think modular, think open, think
abstraction, think service. Build object oriented, service
oriented loosely coupled architectures with micro-services.
Ensure sufficient middleware skills to test infrastructure early.
Infrastructure, cloud and virtualization skills: Test
environments are typically the largest bottlenecks in testing.
Understanding automated environment provisioning, cloud
architectures and environment virtualization, can improve
delivery pipeline by a significant 25% or more.
Native programming languages skills: Coding skills in
native programming is necessary to move away from user
interface (UI) based test automation and to work more closely
and seamlessly with developers.
Developing non UI based automation framework skills,
using proven data driven, keyword driven and hybrid
automation frameworks will be the way forward.
Enterprise level hybrid agile testing skills: Analysis shows
that agile development is being driven by digital programs and
mobile initiatives, with the Scaled Agile framework (SAFe)
methodology taking the biggest market share (31%). Typically
organizations are using a combination of DSDM, SCRUM, XP
noting that SAFe is enterprise driven.
DevOps with Quality can only be achieved by assigning the
central role of DevOps Quality Engineer in each DevOps team.
The DevOps Quality Engineer is a team member who
contributes to development and operations – but with focus
on quality validation. (Some teams use the term SDET
(Software Development Engineer in Testing) for this role. The
DevOps Quality Engineer is never the sole person responsible
for quality – since quality is a shared responsibility of
everybody in the team. The DevOps Quality Engineer plays a
vital enablement role for the whole team to achieve the
required quality standards with each DevOps cycle. The most
important responsibilities of the DevOps QA engineer are:

5
 Figure 4. DevOps Quality Engineer Role

Configuration Server
Infrastructure Application Test Infrastructure
Code Code & Configuration Scripts & Data Code
SCM

Provision / Tear-Down

Provision Commit Commit Trigger Publish

Agile
User Lifecycle
Stories Tracking Build
Tool Checks and Tests INT.

Deployment Workflow

Deploy Binaries Run Tests

Dashboard

Provision Environment
Notifications

IDE IDE
DEV Code Analysis QA Automated CI Deploy
Business Work Unit Testing Work Functional Build Master
Bench Build Bench Testing Slave STAGE
Farm

Provision

Provision
PROD
REPO

Environment
Containers Dependencies Binaries Virtualization Containers

Events Logs Reports Logging & Monitoring Dashboard Trending Analytics

Set up flexible DevOps Quality The commonly used tools for managing and realizing quality in
Automation Framework DevOps environments today are

The enhanced DevOps process requires maximum Components Indicative Tools

automation of all parts of the process. From User Story Test driven development JUnit, xUnit,
gathering to Continuous Development, Continuous Build,
Behavior driven development Cucumber, JBehave
Continuous Deployment and Continuous Monitoring. The
more automation that can be achieved, the more the benefits Exploratory Testing HPE Sprinter
promised by DevOps can be realized. Configuration Management GIT, PVCS, VSS,

A multitude of tools is already available to support these Continuous Integration Jenkins, TFS
activities, and many more will be emerging in the near future. Build automation Make, GNU make, Apache
Test lifecycle and defect HPE QC, Rally, Jira,
A proper set up of the automated quality environment is best
management VersionOne
served by a backbone of corporate agreed tool framework,
with enough freedom to individual teams to plug-in Service automation Parasoft, HPE Service Test,
special tools. CA Lisa
Functional automation HPE QTP, HPE UFT, Selenium
In order to enable continuous improvement of team
performance and implement corporate governance across Performance test HPE Loadrunner
this landscape of tools, there is a need for one corporate level Continuous Monitoring Application, Server,
standardized dashboard of quality metrics. This corporate Database Monitors
dashboard allows for automatic and continuous reporting of Environment virtualization& Puppet, Chef, VM ware ,
source data from all the different tool repositories. Cloud leverage HPE Codar
Service Virtualization CA Lisa
Test Data Management IBM Optim, Parasoft File-Aid

DevOps with Quality 6

 Testing the way we see it

Use a virtualized Environment for on an as-needed basis. The test environments are delivered
DevOps Quality Validation remotely through secure networks, reducing the need for
onsite hardware installation, administration and maintenance.
A flexible and instantaneous DevOps Quality validation
Environment is required to enable right coverage of validation So teams pay for the specific testing and access to dedicated
checks in a repeatable, and cost –effective manner. Such an hardware platforms and industry-leading software licenses
environment is based on three principles: – but only when they need them. Cloud-based Environment
Provisioning is a critical enabler for DevOps Quality Validation.
• Virtualization of Services, Network and Devices
• Cloud Based Environment Provisioning • It provides instantaneous flexibility to switch-on or
• Test Data Management and Generation. switch-off environments
• It provides fully equipped validation environments including
Service Virtualization enables the setup of an environment core test automation tools
in which individual applications functions and/or services can • It reduces complexity, cost and overhead in managing and
be tested as if they were embedded in a real life environment maintaining test infrastructure
with all connections to other systems present. Service
virtualization can be leveraged to record / create all the Test Data Management (TDM), ensures the provisioning
required behavior expected by dependent systems. Through a and management of the test data for all DevOps test sets. Test
standard API set up each Application Under Test (AUT) Data Management offers the ability to (automatically) generate
communicates with the virtual services, instead of the real test data, fast test data sub-setting, test data
systems and this enables teams to perform early integration depersonalization/obfuscation/masking, test data archiving
testing. Service Virtualization reduces the complexity of and test data refresh. In short the test data management
extensive test infrastructures and test data sets. It also activity creates and manages high quality and consistent test
reduces delays that are normally caused if teams need to wait data across all test environments to perform testing activities.
for other teams to complete their services under development. A well managed and automated Test Data Management
Service Virtualization allows a working functional test process is a key enabler to
environment to be made available with much more ease and
at lower cost, and integration defects can be unveiled at earlier • Ensure the timely availability of test data
stages. The increased need for a test environment using • Test data consistency across all test environments
service virtualization is pushed further due to movement to a • Sufficient volume of test data for real-life tests
software architecture based on microservices (small • Predictive test results for test automation
decoupled building blocks that interact with each other • Re-usability of test data sets
through language-agnostic API’s). • Adherence to data security and privacy regulations
• Test data creation/modeling as per business rules
Service Virtualization solutions are offered by key technology
partners such as Hewlett Packard Enterprise, CA, IBM Proposed Roadmap to achieve
and Parasoft. DevOps with Quality
Network Virtualization enables the virtualization of mobile For organizations that are implementing the DevOps
and broadband networks. This allows near real-life testing of philosophies, we recommend using our DevOps QBP
application behavior and application performance in a approach to establish the baseline maturity, roadmap and
controlled test environment. continuous process for improvement in DevOps maturity.

Network virtualization is offered by multiple technology The roadmap to DevOps with Quality will have to cover all the
partners: Hewlett Packard Enterprise (HPE), Citrix, IBM, critical elements
OpenSolaris.
• Integrated QA approach covering Continuous Testing to
Device Virtualization enables the virtualization of devices Continuous Quality Monitoring
such as smart phones, tablets and other devices. It allows • QA awareness and expertise among all disciplines
quality validation with a large number and versions of devices • Quality Engineering role in DevOps teams
immediately and at lower costs. • QA automation
• QA environment and test data provisioning
Device Virtualization is offered by multiple technology partners:
PerfectoMobile, VMWare, Xamarin, MobileLabs and It is imperative to create business cases and DevOps driven
Red bend. roadmaps with right lean metrics aligned to the DevOps
business objectives. This means shifting from the traditional
Cloud-based Environment Provisoning. With cloud based
“cost of quality business cases” to “cost to achieve desired
environment provisioning it is possible to switch on or switch
test velocity business cases”.
off personalized test environments with test automation tools

7
To find out how Capgemini and Sogeti’s Testing Services can help your organization achieve its Testing and QA business
goals, please contact your local Capgemini or Sogeti testing representative or our Global Testing Services Sales Team:

Mark Buenen Renu Rajani

VP Global Service Line Testing VP Global Service Line Testing
[email protected] [email protected]

Deepika Mamnani Steve Hughes

Principal | Financial Services GBU Senior Vice President | Testing Practice Manager, Sogeti USA
[email protected] [email protected]

About Capgemini and Sogeti

With more than 180,000 people in over 40 countries, Capgemini is one of the world’s foremost providers of
consulting, technology and outsourcing services. The Group reported 2015 global revenues of EUR 11.9 billion.
Together with its clients, Capgemini creates and delivers business, technology and digital solutions that fit their
needs, enabling them to achieve innovation and competitiveness. A deeply multicultural organization, Capgemini
has developed its own way of working, the Collaborative Business ExperienceTM, and draws on Rightshore®, its
worldwide delivery model.

Sogeti is a leading provider of technology and software testing, specializing in Application, Infrastructure and
Engineering Services. Sogeti offers cutting-edge solutions around Testing, Business Intelligence & Analytics,
Mobile, Cloud and Cyber Security. Sogeti brings together more than 23,000 professionals in 15 countries and has
a strong local presence in over 100 locations in Europe, USA and India. Sogeti is a wholly-owned subsidiary of
Cap Gemini S.A., listed on the Paris Stock Exchange.

For more information, please visit:

www.capgemini.com/testing or
www.sogeti.com/testing
MCOS_GI_AH_20160506

©2016 Capgemini and Sogeti. Rightshore ® is a registered trademark belonging to Capgemini.

TMap®, TMap NEXT®, TPI® and TPI NEXT® are registered trademarks of Sogeti.

No part of this document may be modified, deleted or expanded by any process or means without
prior written permission from Capgemini.