Measuring The Effectiveness of An Internal Control System: by Dr. David Brewer and William List, CA, Hon FBCS
Measuring The Effectiveness of An Internal Control System: by Dr. David Brewer and William List, CA, Hon FBCS
Page 2 of 28
BACKGROUND
The Need for Control
Ever since organisations expanded beyond the
control of the "boss" there has been a need for
controls to regulate their activities. For
example the profession of accountancy/audit
grew out of the need for owners to check on
their factors/agents overseas in the 19th
century. As private companies expanded and
brought in outside shareholders (joint stock
companies) the need to regulate the behaviour
of those running the companies grew and the
first set of legislation governing companies
was passed in the early 20th century.
Since the Second World War there has been
very substantial change; the development of
IT, the expansion of cheap communications
(both travel and telephones) across the world
etc. These new facilities have been harnessed
by commerce to create world wide
organisations that can be operated from one
point on the globe. The need therefore to
update the legal framework for the conduct of
commerce (and governments, charities etc)
was recognised and a large volume of laws and
regulations now exist in most countries
specifying standards of conduct and controls
that must be complied with by organisations.
Many of the new laws are a result of scandals
where it was perceived that the investing
public (directly or through co-operative
investments) were being "ripped off" by the
inappropriate conduct of senior executives.
One only has to consider the South Sea
Bubble, Kruger, Salad Oil company, Equity
funding, Polly Peck, Maxwell Pensions, Enron,
Gamma Secure Systems Limited,
Corporate Governance
Operational Risk
In particular we are concerned with the
processes to limit operational risk within an
organisation. At present the financial services
regulators world wide are seeking to change
the processes within the regulated
organisations to accord with the Bank of
International Settlement's (BIS) requirements
set out in BASEL 2. National regulators and
BIS are issuing guidance on the
implementation to regulated organisations.
Page 3 of 28
Types of Control
We assert, for the purpose of explaining our
theory, that a risk materialises on the
occurrence of an event, the consequences of
the event being the damage caused by the
adverse impact (and recovery from that
impact). There are three classes of controls:
Gamma Secure Systems Limited,
Our Objectives
The problem facing the senior management
with regard to the controls can be expressed as
the following questions:
Do the controls work (including are they
performed correctly)?
Are they cost effective?
Do we have sufficient (neither too many or
too few)?
Organisations monitor their controls in two
main ways:
Investigating incidents (i.e., events and
impacts) and making amendments to
controls as appropriate
risk
Over-reliance on technology
Page 6 of 28
Page 7 of 28
Acceptable Risk?
The Audit Practices Board (APB) presents an
interesting example of acceptable risk.
Basically, the example concerns a small
advertising agency. Small adverts are placed
for cash and the company accepts the risk the
5,000 worth of cash transactions may be lost
per annum, for whatever reason. The APB
example argues that the cost of the controls
necessary to assure each transaction would be
disproportionate to the value of the
transactions. The problem of such loses is
subsequently ignored.
Our question is "How does the company know
when the loss becomes 5,001?" Surely, that
ought to be an unacceptable risk!
Observations
What the APB example fails to argue concerns
when this acceptable risk becomes an
unacceptable risk, i.e. when the loss becomes
5,001. First, of course, you need a way to
determine when it does. A reconciliation, each
month, of the cash received versus the
advertisements would serve this purpose. It
would highlight the total loss, albeit being
unable to identify the particular transactions
concerned. However, it is the total that we are
interested in at this stage of control.
If the reconciliation, performed at the end of
month 11, shows that the loss is 4,580 then
the loss remains acceptable (as it is just on
target to come under 5,000) and the company
can be satisfied with its decisions. If the same
loss in reported at the end of month 1, then the
company ought to be concerned that its
acceptable loss is in danger of becoming an
unacceptable loss in month 2, and ought
therefore to take action accordingly. Once
again, it is necessary for the ICS to detect the
event (in this case the metamorphosis of
Over-reliance on Technology
At a meeting, our client's IT manager asked
why his networks had just been the victim of a
well known virus. We asked some questions
and sent him off to find the answers. During
his absence, a colleague remarked that for
some time his laptop had been reporting that its
anti-virus library was not up-to-date. Others
quickly reported the same. The IT manager
reported back. Anti-virus library upgrades were
being received in a timely manner by the
server but due to a software problem they were
not being distributed to any other computer on
the network. The software had stopped
functioning 3 months ago!
With another client, we asked some questions
to determine whether the anti-virus libraries
were up-to-date. They were, save for all the
directors' laptops. Further investigation
revealed that they were scheduled for a regular
update every day at 05:30. No director had
ever docked their laptop at that the unearthly
hour in the morning. Their libraries were two
years out of date! We asked about their new
web-surfing controls. The QA manager, a
railway model hobbyist, proudly announced
that it prevented him access to his hobby sites,
and having been denied once he had never
tried again. We asked him to try once more,
and guess what - he had access. The software
had stopped working.
Observation
These stories remind us that controls do not
always work as intended and from time to time
they fail, but does anyone ever check!
Page 9 of 28
Summary
Our observations in respect of each of these
stories have much in common. They are:
Without loss of generality, an ICS must
detect the event in sufficient time for
something to be done about it. (See BA122,
Chip and PIN, Acceptable risk and
Software Intensive Projects.)
FUNDAMENTAL MODEL
In this section we introduce our Fundamental
Model. Let us start by supposing that an
organisation carries out a range of business
activities. Let the cost of such activity be CBA.
Cost may be expressed in terms of money and/or
resources (e.g. volunteer work). It will generate
some business benefit B. If the organisation is a
company, then B corresponds to profit, P, and is
related to the cost of the business activities
through revenue R:
P = R CBA
The organisation deploys an Internal Control
System (ICS). This has an associated cost, CICS,
which increases the cost of doing business
CBA + CICS
In the context of a company this has the effect
of reducing profit, see Figure 1.
Page 10 of 28
Page 11 of 28
CLASSES AND
CATEGORIES
Control Classes
We define seven classes of control, see Table
1. They fall into three broad categories of
control, traditionally known as preventive,
detective and reactive. Class 1 is higher than
Class 2, etc.
Class Ability to detect the event Type
Preventive
Detective
Control Failures
It is important to recognise that all controls
may fail, as exemplified earlier (see page 9).
Page 12 of 28
Self-Policing Procedures
The defence is to have some other control to
address failures in the first. In practice there
will be a sequence of controls as illustrated in
Figure 7.
Page 13 of 28
Robustness
Speed
The middle ground criterion is:
S1 - There is a mixture of Class 2, 3 and even
Class 4 detective controls. The Class 2 and 3
controls that are not protected by fail-safe selfpolicing procedures may degrade to Class 4.
Anticipation
The middle ground criterion is:
A1 - There is at least one Class 6 BCP dealing
with some catastrophe (e.g. fire). Other
unexpected events incidents are dealt with
through an ad hoc procedure.
A stronger criterion is:
A2 - There are a variety of BCPs (some of
which may be Class 5) dealing the failure of
control or some catastrophe (e.g. fire). Other
unexpected events incidents are dealt with
through an ad hoc procedure.
Categories of ICS
We can apply the criteria and determine the
category of the ICS using a simple marking
scheme. We award 3 marks for each of R1, S1
and A1 and award 1 extra mark if it is
exceeded.
The resulting categorisation is:
Well above average (AAA rating) 11 or
higher
Above average (A*) 10
Average (A) 9
Below average (B) 6 - 8
Well below average (C) 4 or lower.
Example 1
To achieve a AAA rating, we need to satisfy all
three criteria and surpass at least two. Thus we
gain 3 marks for each criterion that is satisfied,
giving 3 x 3 = 9, plus 1 mark for each criterion
exceeded, giving 9 + 2 = 11. If we exceed all
three criteria, then the total mark is 9 + 3 = 12,
i.e. for AAA rating we need to score 11 or
higher.
Example 2
To achieve a B rating, we fail one criterion, but
we might exceed either or both those that we
pass. We achieve 3 marks for passing a
criterion and 4 if we exceed it. Thus, for a B
rating, at worst we just pass two (i.e. the total
mark is 3 + 3 = 6) and at best we exceed two
(i.e. the total mark is 4 + 4 = 8). If we pass two
and exceed one, the total mark is 3 + 4 = 7.
Thus the range of marks that give a B rating
are 6 - 8.
OPERATIONAL
EFFECTIVENESS
Objective
Management needs to know whether or not the
current ICS, i.e. the one actually in place and
working now, is achieving the objectives they
want, irrespective of what else is happening in
the world. In other words, the measurement
should not be conditional on whether or not
anyone is trying to attack the organisation or
Page 15 of 28
Measuring
Measurement of operational effectiveness is
straightforward. It takes the form of:
Determining the actual control class of
each control, using Table 2
Applying the criteria specified in Table 3.
COST EFFECTIVENESS
Objective
Operational effectiveness does not necessarily
imply cost effectiveness. To determine the cost
effectiveness of the ICS we need to apply other
metrics, e.g. CICS, as identified in the
fundamental model.
Measurement
A Worked Example
Consider a small software company that
produces bespoke software system for its
clients. The company relies on an ICS that is
predicated solely on program testing. In
particular, there are no formal design/code
reviews. There is a reliable backup system that
verifies that backups are restorable and
complains if they are not. However, there is no
BCP covering anything outside of IT. What is
the operational effectiveness of this approach?
A typical development schedule is shown in
Figure 8. The "program testing" control takes
effect late on in the schedule. It may start to
identify problems as early on as month 6, but
some problems might not be detected until
month 12. If the control identifies a top level
design error then the later it is detected the
greater the chance that it will be too late to do
anything about it before the expiry of the time
window, which we will associate with the end
of the development period. Thus, the "program
testing" control is Class 2, potentially
downgrading to Class 4. It does not tell you if
it fails to find an error and therefore it is not
self -policing. The backup control, however, is
self-policing but is not fail-safe.
Page 16 of 28
Detective Controls
# Question
less than the reduction in impact penalty, minus
the loss in business benefit, multiplied by the
number of times the BCP might be invoked in
that period of Y years?" If the answer is yes, then
pre-deployment is worthwhile.
Preventive Control
Should we be using a preventive control? The
answer is likely to be yes if the cost of using a
preventive control is less than the sum of the
cost-to-fix and possible impact penalties for
all the events that the preventive control is
designed to detect, i.e.
Cpreventive control < all events, j, that the preventive control
detects (CFj + IPj)
The cost of using a preventive control includes
the cost to buy/develop, install, configure,
commission, operate, train people in its use,
audit its use and maintain it. We have not
included in the above formula the cost of the
controls that would otherwise perform the task
of the preventive control as we recommend
that they be retained in case of a control failure
in the preventive control. An impact penalty
will only occur if the corresponding existing
controls are Class 4 or lower.
Note the summation over all the events that the
preventive control is designed to detect. In
practice, this number will be an estimate as it
concerns the future. If the cost-to-fix and
possible impact penalty for each event is
constant (or can be considered approximately
so), the inequality becomes:
C preventive control
<
N * ( CF + IP ) average
Page 17 of 28
N * IP average
Pre-deployment
If we pre-deploy all or part of a BCP there will
be an associated pre-deployment cost and a
maintenance cost. Pre-deployment costs may
include equipment purchase/lease, building
purchase/hire, insurance, extra staff, training,
commissioning, regular tests and practices, etc.
Prior to invoking the BCP there may also be an
associated business benefit, BBCP, which will
offset the pre-deployment costs, Cpre-deploy. For
example, a redundant IT installation, kept in a
IPreduction - BBCPloss
BCP Need
Do we need a BCP? The first question to ask
is what would be the impact penalty if the
unthinkable was to happen? Having answered
that question, the next question is "is that an
acceptable risk". If it is not, then you need a
BCP.
Surprisingly, in response to the second question
the answer "if I am still alive, I'll just start up
again" is a Class 6 control. The question is then,
how much of this plan should we pre-deploy
(clearly following carefully consideration and
suitably refinement of this somewhat
embryonic/flippant BCP!).
A Worked Example
(continued)
The foregoing allows us to answer general,
albeit important, questions about the
effectiveness of our ICS. By way of our
worked example, however, we look at
something very specific and entirely in tune
with what the focus of cost-effectiveness for a
commercial organisation ought to be. i.e.
profit.
Let us suppose that our small software
company is being invited to bid for a new
project. The contract is for a fixed price with
stage payments. The development is scheduled
to last one year. The final payment is due on
satisfactory completion of the project. If there
is an overrun, there is an impact in the form of
a revenue penalty which increases with the
extent of the overrun. The agreed schedule is
shown in Figure 8.
Page 18 of 28
ICS#1
ICS#2
ICS#3
ICS#4
Yes
(5.5)
6.8
9.8
8.8
No
9
8.3
10
9
Table 5: The bottom line effectiveness of the
four candidate ICS (fixed price)
The company decides upon ICS#3.
An alternative scenario
It is interesting to consider what might happen
if the contractual situation was quite different.
What would happen, for example, if the
contract was time and materials and there was
no penalty clause. Suppose the company elects
to charge its analyst programmers at a daily
rate, equivalent to 1.67 MU per month, and its
chief analyst programmers at a daily rate,
equivalent to 2.09 MU per month. These rates
allow for overhead and profit.
Ignoring maintenance, as that being on a time
and materials basis, it would appear that the
greater the number of bugs, the more
maintenance work is required and therefore the
greater the profit(!), we have:
ICS
Revenue
ICS#1
60
(no event)
ICS#1
70.1
(event occurs)
ICS#2
66.8
(no event)
ICS#2
69.3
(event occurs)
ICS#3
60
(no event)
Cost
51
Profit
9
59.5
10.6
56.7
10.1
58.2
11.1
53
Page 19 of 28
Page 20 of 28
Comment
Just who takes the risk in these situations is an
important decision. For ICS#1 (see Table 7
above) the client takes all the risk. If the
company (i.e. the contractor) makes an error,
the client pays for it. With ICS#2 -4, the
company has a better ICS, which in Cases 2
and 4 does cost the client more. However, in
all cases the client pays less than in Case 1 if
the company does make an error. The question
therefore is "is it in the client's best interest to
pay the company to improve the effectiveness
of its ICS?" Table 7 suggests that the answer is
not only "yes" but also that it can be done for
next to nothing (compare ICS#3 with ICS#1).
Perhaps this example gives us an insight as to
why customers do put pressure on large
suppliers to introduce management systems
(whether for risk, or subordinate areas such as
quality or information security).
MEASURING
IMPROVEMENT
Objective
If the current ICS is not achieving
management's objectives, management needs
to be able to determine what needs to be
changed and plot a course of action to
implement those changes. As the ICS evolves,
management again needs to measure the actual
ICS to see how its effectiveness is improving.
Apart from the overall ICS, management may
focus attention on particular controls. This
may be in response to incidents or changes in
threat.
Measurement
There are two types of improvement:
An improvement to the overall ICS
An improvement to an individual control.
Measurements are generally made using the
methods previously described, i.e. using Table 2
to determine the actual class of a control, using
Table 3 to determine operational effectiveness
and Table 4 to determine cost-effectiveness.
However, individual
Improvement to an individual
control
There are a variety of improvements that you
may wish to make to an individual control, or
group of controls, and conform by
measurement, e.g.
Advancement to a higher Class or
within Class
A Worked Example
(continued again)
Let us return to our software company example
and this time consider the candidate ICS
identified when we were considering the costeffectiveness problem. The category for ICS#1
is A, as determined previously. The chosen ICS
(#3) adds a new control, which is Class 2 nondegradable. By itself this is insufficient to
change the category of the ICS. However, it
was chosen because it ought to increase the
cost effectiveness of the ICS. We need to
monitor the cost of the control, the cost of fix
and any impact penalty to confirm this. We can
also monitor the improvements in the control
itself by recording the time to detect and the
time to fix for the events that it discovers.
Other questions that we ought to monitor are:
Are there errors that it was designed to
detect that escape detection?
Are these trapped by the software
testing control?
What proportion of these escape detection
and are ultimately found by the customer?
RISK TREATMENT
PLANS
In this section we propose a methodology for
generating Risk Treatment Plans, which makes
use of the fundamental theory which we have
just discussed.
We have used the methodology extensively in
the information security arena. We have taught
senior managers/risk owners how to use it in
various parts of the world and they have been
able to apply it, not only in the context of
information security but also to other
business/governance concerns. We therefore
believe that this methodology works and is
teachable, repeatable and reproducible.
We start by saying a little more about events
and impacts on which the methodology is
founded.
Page 22 of 28
Events
The events referred to in this paper are bad
things that cause trouble. The insert (below)
lists those events, which in our BS7799 work
we feel are common across many businesses.
In addition we would add other events that
were specific to that particular organisation.
An example would be "one of our aircraft
has broken down in the Indian Ocean". We
told a story about this on page 4.
Theft
Acts of God, vandals and terrorists
Regular fraud
IT failure
Hacking
Denial of Service attacks
Disclosure
Breach of the law
Impacts
Customer dissatisfaction
Adverse press coverage
Loss of revenue
Unanticipated costs
Inability to carry out some or all of its business
Loss of the monetary value of buildings
and contents
Failure to prosecute
Court action against an employee or the business
itself
Page 24 of 28
CONCLUSIONS
This paper has set out:
the methodology to measure the
effectiveness of the control element of an
ICS
Page 26 of 28
ICS Metrics
We propose two sets of metrics for use in
determining the effectiveness of an ISMS
within an organisation. The first set of metrics
is independent of external factors and is
therefore a true measure of the effectiveness of
the organisation's procedures and management
system.
Operational effectiveness is determined
solely by measuring the time parameters.
Classes of ICS
Applicability
Page 27 of 28
Acknowledgments
We would like to record our thanks to the very
many people, all over the world, who have
over the years taught us about Information
Security and Internal Control structures.
In particular in respect of this paper we would
record our thanks to
Matthew Pemble of RBS
Richard Hackworth of HSBC
David Spinks of EDS
Harvey Mattinson of the UK Cabinet Office
Michael Nash of Gamma Secure Systems
Leslie McCartney of Reuters
Ted Humphries of Xisec
Marc Kekicheff of Visa International
Regina Brewer of Konami (Europe)
We trust that our contribution will be seen as
our thanks to them for their training to us.
References
[APB guidance] Briefing paper - Providing
Assurance on the effectiveness
of Internal Control issued by
the Audit Practices Board July
2001, see
http://www.apb.org.uk/ Copies
from ABG Professional
Information
[Higgs]
[email protected]
[BASEL 2]
[BS7799-2]
[ISO 14001]
Environmental management
[ISO 9001]
[OECD]
[SarbanesOxley]
[Turnbull]
Both are currently part of the international team developing the 7799 family of standards, and are two
of the driving forces behind the Part 2 ISMS standard. They provided training in implementing
ISO/IEC 17799 and have assisted many clients to build ISMSs since 1998 in Europe, East Africa and
the Far East.
Page 28 of 28