Professional Documents
Culture Documents
Model Risk Tiering
Model Risk Tiering
DOI: 10.21314/JRMV.2019.205
Copyright Infopro Digital Limited 2019. All rights reserved. You may share
using our article tools. This article may be printed for the sole use of the
Authorised User (named subscriber), as outlined in our terms and condi-
tions. https://www.infopro-insight.com/termsconditions/insight-subscriptions
Research Paper
Promontory Financial Group, 801 17th Street NW, Suite 1100, Washington, DC 20006, USA;
emails: [email protected], [email protected], [email protected],
[email protected]
ABSTRACT
Model risk management continues to be an important area of risk management across
financial services firms. Regulatory expectations for model risk management apply
to an expanding number and variety of financial firms. In our conversations with
clients, we find that many risk managers are unaware of standard practices in devel-
oping and implementing frameworks for model risk management. This paper – based
on our experience as consultants, former banking supervisors and leaders of units
focused on model risk management – seeks to shed light on one critical area of such
frameworks: model risk tiering, or the rating of risk inherent in the use of individual
models, which can benefit a firm’s resource allocation and overall risk management
capabilities. Although the range of practice in model risk tiering across financial
services is broad and varied, our work has revealed a number of insights.
Most financial firms use explicit procedures and decision tools to assign
models to risk tiers.
Nearly all firms assign models to several risk tiers rather than seeking to rank
them individually from high to low.
47
48 N. Kiritz et al
Although most firms use model risk-tiering tools, these tools serve primarily
to systematize the application of judgment to the risk posed by models rather
than to directly quantify risk.
Tiering tools generally fall into one of two categories – scorecards or decision
trees – with scorecard approaches being the more common of the two.
Tiering tools generally take into account both the risk and the impact of model
failure; the building blocks for these two factors include a variety of descriptors
regarding the construction and use of each model in the inventory.
Keywords: model risk tier; model risk rating; SR 11-7; OCC 2011-12; supervisory guidance on
model risk management; model governance.
1 CONTEXT
1.1 Model risk
Modeling continues to play an increasing role in financial services. Indeed, as more
and more data becomes available and the technology for using data becomes cheaper
and more powerful, more sophisticated quantitative modeling and business analytics
are spreading across the industry landscape. As modeling processes come to play
a larger role in business decisions, the need for model risk management continues
to grow. Financial regulators are increasingly recognizing that need, incorporating
model risk management requirements into rulemaking across an increasing variety
of financial regulations.
Model errors continue to drive unexpected losses. Some of the more spectacular
model risk failures have been well documented. High-profile examples include Long-
Term Capital Management in 1998, the evident shortcomings due to using Gaussian
copulas in the pricing of structured products in the financial crisis, and the AXA
Rosenberg loss.
Bulletin 2000-16 from the US Office of the Comptroller of the Currency (2000),
which focused on model validation, was the first formal supervisory guidance on
model risk for financial firms. Model risk was subsequently addressed at a high level
in the Basel capital rules, developed in greater depth by the Committee of European
Banking Supervisors in its guidelines on the implementation, validation and assess-
ment of the advanced measurement and internal ratings-based approaches (Commit-
tee of European Banking Supervisors 2006), and expanded in the European Union’s
Solvency II rules for insurers (European Parliament and Council of the European
Union 2009). Globally, the most important collection of supervisory documents on
model risk management (MRM) to date is the US regulators’ consistent, but sepa-
rately issued, Supervisory Guidance on Model Risk Management (Board of Gover-
nors of the Federal Reserve System 2011; Office of the Comptroller of the Currency
2011; Federal Deposit Insurance Corporation 2017: collectively known as the US
banking agency guidance).
Model risk has been addressed within a number of other supervisory and regula-
tory documents. These documents include the Capital Requirements Regulation in
the EU (European Parliament and Council of the European Union 2013), the Model
Risk Management Guidance of the Federal Housing Finance Agency (FHFA; Fed-
eral Housing Finance Agency 2013), and the Securities and Exchange Commission’s
(SEC’s) Standards for Covered Clearing Agencies (Securities and Exchange Com-
mission 2016) and Nationally Recognized Statistical Rating Organizations (Securi-
ties and Exchange Commission 2014). More recently, the Office of the Superinten-
dent of Financial Institutions in Canada released a guideline titled “Enterprise-wide
model risk management for deposit-taking institutions” (Office of the Superintendent
of Financial Institutions 2017).
1 Although the expanded coverage of MRM driven by the US banking agency guidance likely
drove the proliferation of model risk-tiering processes across the industry, it had already been
enhanced in a number of firms, including Fannie Mae, where one of this paper’s authors developed
a risk-tiering scorecard to rationalize the massive model validation effort undertaken to support the
financial restatement in 2005–6.
2 LITERATURE
Little academic, technical or regulatory literature has attempted to address the pro-
cess of tiering the contents of a model inventory by the risk posed to an organization.
Although model risk tiering was implemented by some banks after the release of
OCC 2000-16 (or even before), neither that bulletin nor the subsequent US bank-
ing agency guidance explicitly addresses model risk tiering. In contrast, the FHFA’s
guidance (Federal Housing Finance Agency 2013) explicitly discusses risk ranking:
Policies and procedures should provide clear guidelines for developing model classi-
fications or risk rankings. A model’s complexity, business impact and the extensive-
ness of use generally determine its risk ranking or classification. In large, complex
enterprises, the model risk management group and internal audit should harmonize
their model risk rankings to achieve consistency.
(discussed further in Section 4.1).2 Keenan (2015) similarly addresses model risk
ranking, although in the context of a broader discussion of modeling and model risk
measurement. While his goal is clearly stated as risk quantification, Keenan main-
tains that “the first order goal of the framework is rank ordering of models by risk”.
Although Keenan’s objective is to build a “model risk score” (MRS) to rank-order
individual models rather than just assign them to risk tiers, his approach could be
used for tier assignment as well. Keenan first proposes to build an “intrinsic model
risk score” that combines measures of the risk of technical model failure with mea-
sures of operational risk, and with a “nesting penalty” that penalizes models that use
other models’ outputs as input components. The intrinsic model risk score is added to
a “risk mitigation index”, which is meant to reflect the quality of controls applied to
the model in order to yield a “model quality score”. This score is, in turn, added to an
exposure measure to yield the MRS. Keenan uses Figure 1 to illustrate his approach.
Industry surveys provide additional public information about current practices in
model risk tiering. As an example, the American Bankers Association has published
multiple surveys on MRM in cooperation with consulting firms, the most recent of
which was released in October 2017 (American Bankers Association 2017). Useful
information that may be found in industry surveys includes the types of criteria used
in risk-tiering tools, areas with responsibility for model risk tiering and the aspects
of MRM that commonly vary by model risk tier.
accept the necessity of a significant role for expert judgment and heuristics;
avoid unnecessary complexity to ensure that the tiering process – and the data
and tools that support it – is tractable;
2 Mankotia and Joshi (2013) point out that “model mechanics and [measuring risk for] specific
types of models have garnered most of the industry’s attention”. They are unable to find “any
existing model risk measurement methodologies that can be applied to a broad range of models or
that are comprehensive in nature”.
tier models relative to the rest of the firm-wide inventory, not relative to smaller
business or functional areas, to promote consistency in the tiering process
across the entire model inventory within the organization;
aim for a reasonable distribution of model assignments across risk tiers; and
Most of these principles stem from the quantitative challenges inherent in the pro-
cess of assessing model risk and the critical role that such assessment plays in risk-
sensitive MRM frameworks. Model risk-tier assignments drive the frequency and
intensity of many activities. For that reason, a model risk-tiering tool must be widely
accepted and understood. Model risk managers use model risk tiers to schedule vali-
dation activities and determine their scope. Model owners use model risk tiers to help
inform the model approval process. Senior management relies upon model risk tiers
to receive clear and concise model risk reporting. Therefore, model risk-tiering tools
must be simple to use and logical for both technical and nontechnical audiences.
Any criteria underlying the risk-tiering system should be lucid and any calibrations
intuitive. A firm-wide awareness of any model risk-tiering tools in use encourages
a strong risk culture and supports the consistent assignment and interpretation of
model risk tiers.
The remainder of this section discusses these principles in more detail.
3.3 Transparent
Although expert judgment necessarily plays an important role in most cases, prac-
titioners also try to use objectively measurable inputs wherever practicable in order
to enhance transparency and consistent application. Having the process be transpar-
ent within an organization is important, not least because it increases organizational
acceptance of the risk-tiering framework. Transparency also reduces uncertainty
about model tier assignments, making it easier for model owners and developers
to orient their development and documentation efforts toward meeting the standard
required by the risk tier to which they expect their models to be assigned. However,
some important indicators of model risk, such as model complexity, are exceedingly
difficult to measure objectively. Developers of risk-tiering tools must balance the
goal of transparency with that of incorporating all significant model risk drivers,
since some risk drivers will likely be judgmental.
In order to establish transparency in the development of model risk-tiering tools,
developers often solicit input from significant groups of model owners, developers
and users, as well as from senior management. In addition to fostering transparency,
such an open development process also serves to enhance the organizational legiti-
macy of the resulting risk tiers as well as to promote consistency in model risk-tier
assignments across the organization.
Most firms delegate development and implementation of the risk-tiering method-
ology to the MRM group, with the head of MRM retaining the authority to override
the results of the framework should they appear inappropriate for certain models.
Some policies only give the head of MRM the authority to move individual models
to a higher (but not a lower) risk tier than that resulting from the risk-tiering tool.
3.4 Tractable
Model risk-tiering tools should be tractable, that is, easily managed and adjusted.
Significant factors that affect tractability include the ease or difficulty of obtaining
or generating data on the model properties used for risk tiering; the simplicity and
flexibility of the risk-tiering process and any internal calculations it uses; and the
ease with which the tool can be tuned to bring results more closely in-line with
the expert judgment on which the tiering relies. The ability to modify the tool to
reflect expert judgment is generally more important than, for example, incorporating
complex processes for inputs or rankings in a quest for precision. More tractable tools
also tend to be more transparent, so the principles of tractability and transparency
reinforce one another.
3 The Committee of Sponsoring Organizations of the Treadway Commission (2004) states: “Inher-
ent risk is the risk to an entity in the absence of any actions management might take to alter either
the risk’s likelihood or impact. Residual risk is the risk that remains after management’s response to
the risk. Risk assessment is applied first to inherent risks. Once risk responses have been developed,
management then considers residual risk.”
4 This may be particularly important when model risk-tiering systems are first implemented. Many
firms first implement a risk-tiering system while working through a model backlog, and the ini-
tial model inventory may include many unvalidated models. This creates significant challenges for
approaches that include validation results in risk-tier assignments, as the vast majority of mod-
els may lack necessary data; Keenan’s (2015) proposed framework, for instance, places critical
reliance on model validation and ongoing monitoring results.
5 We note, however, that this definition of use can cause some confusion where firm models are
used to provide financial advice to firm clients or made available for public use. While firms may
not base their own business decisions on these models, their clients or the general public might.
Failure to reasonably control such risk may be viewed by regulators such as the SEC as failures in
fiduciary duty and result in financial penalties and reputational harm.
posed by other models in the inventory. The metric produced thus represents rela-
tive, not absolute, risk. Moreover, that risk is relative to other models in that specific
firm’s inventory, and not relative to the models in use at other firms. This has impli-
cations for the definitions of risk-tiering factors and any weights placed on them.
For example, an exposure threshold for a regional bank with $1 billion in assets will
almost certainly differ from that used by a systemically important bank with $1 tril-
lion in assets; if not, then one firm would end up with a much higher proportion of
high-risk-tier models than the other. The differences in size and complexity between
firms are addressed by differences in the MRM frameworks, which typically dictate
more robust MRM activities at each tier for larger and more complex firms than for
smaller, simpler ones.
6The terms “model profile” and “model profile elements” refer to data about a model and exclude
data used in the development or operation of the model.
the part of model owners, developers and risk managers, while reassigning a model
to a lower risk tier may be perceived as reducing the value of similar work already
completed. A process that results in frequent revisions to tier assignments after model
implementation can significantly undercut internal acceptance of the entire model
governance framework.
A desire for stability in tier assignments does not override all other considerations.
The risk tiers assigned to models can, and in some cases should, change over time
if new information about model risk comes to light, or if a firm identifies a need to
revise the MRM framework. The assignment of a model to a different risk tier can
be caused by changes to thresholds or other aspects of the model-tiering tool, or by
changes in the model profile elements utilized by the tool. This latter change may
be caused by the discovery of additional uses, the correction of data errors or the
reevaluation resulting in changes to subjective elements of the model profile.
Firms generally find that the stability and predictability of model risk tiering
increases over time as staff gain more experience with a given tool and its inputs,
and as awareness of the tool spreads within the firm. The key to establishing a sta-
ble and predictable model risk-tiering tool appears to be a methodical development
process that provides maximum transparency regarding both the tool’s methodology
and its results to all stakeholders as development progresses.
4 OBSERVED METHODOLOGIES
We observe a wide range of risk-tiering methodologies in practice. However, they
largely fall into two broad classes. Most firms opt for some variation of a scorecard.
A few use a decision tree approach.
Exposure level
Exposure Level Exposure level
Expert overlay: are there other significant considerations that would justify overriding the preliminary risk rating?
Final risk rating
to the high or moderate risk tier, depending on notional exposure addressed by the
model relative to a threshold; if not, the model is assigned to the low or moderate
risk tier, again depending on notional exposure relative to the threshold. Later boxes
rely on judgment regarding the model’s use and exposure level.
As may be evident from the description, this particular decision tree implementa-
tion relies heavily on judgmental inputs. It also relies on binary categorical outcomes
and therefore does not allow for more than two answers to any question. This is a
common approach to decision tree construction both because multiple categorical
outcomes (also known as multiple-level categorical variables) can make decision
trees very complicated very quickly – and thus difficult to represent and interpret –
and because decision trees may become biased with the inclusion of categorical vari-
ables with multiple levels (see Deng et al 2011). Visual representation of decision
trees can get even more challenging when multiple-level categorical variables and
larger numbers of variables are combined.
In practice, in part due to these challenges, designers tend to keep decision trees
simpler than scorecards, even though very complex decision tree models have been
developed in various disciplines (see, for example, Lewis 2000). While simple is
good, oversimplification can create problems for a tiering process. For instance, in
the uppermost diamond of Figure 2, the use of a binary variable identifying whether
the model measures risk, price or value may create a very strong binary effect, where
a more continuous multiple-level categorical variable would give a more nuanced
result that aligns better with expert judgment. In practice, such binary materiality
variables can create tools that inadequately differentiate model materiality and result
in a concentration of model assignments in higher tiers (clustering).
4.2 Scorecards
Figure 3 shows a sample model risk scorecard (MRS). This scorecard uses a factor/
component approach with four factors. Some factors are represented by single com-
ponents while others utilize multiple components. The “value range” column gives
potential values for the model inputs, some of which may be directly sourced from
the model profile while others may require preprocessing to convert variables with
continuous or widely varying values into more limited categorical variables. Exam-
ples of variables that must be converted to categorical types include estimated dol-
lar exposure, number of users and number of input variables. The “weight” column
gives the percentage weight of each factor (in yellow) in the final score, along with
the weight of each component of the factor value (in pink). In addition to the four fac-
tors, this MRS includes two adjustments: a “capital stress testing” indicator, which
can push a model into tier 1 regardless of its score on other factors, and an MRM
adjustment, a score override allowing the head of the MRM group to add up to 50%
of the total possible value to the base metric. The MRS produces a final risk score
to drive the risk-tier assignment, as seen at the bottom of Figure 3. The process of
setting preprocessing thresholds, component and factor weights, and total risk-score
tier-assignment thresholds is addressed below in our discussion of calibration.
The sample MRS in Figure 3 is clearly more complex than the sample MRT in
Figure 2. It mixes binary variables with multiple-level categorical variables. It also
incorporates a weighting scheme that may be less intuitive than that implicitly incor-
porated into the decision tree structure. However, the MRS has other advantages. For
firms that wish to include several categorical variables with multiple levels, the MRS
will likely prove easier to represent visually, and thus easier for users to understand.
To understand how this scorecard works, consider the calculation of the “oper-
ational factor”. This factor has a weight of 10% in the final score, with 50%
Adjustments:
Pillar I/II input 0 or 1 Override 0 1 0 0
MRM adjustment 0–50 50% 0 0 10 0
Model scores
Materiality factor 15 22.5 22.5 30
Regulatory exposure factor 30 30 0 30
Operational factor 7.5 2.5 7.5 7.5
Model complexity 16.5 3 3 19.5
Capital stress testing 0 1 0 0
MRM adjustment 0 0 10 0
Total score 69 59 43 87
Model tier Tier 2 Tier 1 Tier 3 Tier 1
www.risk.net/journals
Model risk tiering 63
5 DESIGN
Tiering tools are often developed as part of the initial development or overhaul of
the larger MRM framework. We see a number of important preconditions for a
successful development process, including the following:
close coordination between developers of the risk-tiering tool and the model
owners in order to establish clarity around profile-element definitions and
challenges in profile-element collection; and
good communication across the firm regarding principles for developing the
model-tiering tool and evolving tier assignments.
The choice between an MRT and an MRS – tree versus scorecard – is a key deci-
sion that determines other aspects of the design process, since some variables will
be more amenable to one approach than to another. As noted above, a smaller num-
ber of binary variables are likely better suited to an MRT, whereas a larger num-
ber of categorical variables with multiple levels will be better suited to an MRS.
Nevertheless, it is possible to develop the variables first and then to choose a tool
design that accommodates the selection and availability of the chosen variables.
5.1.1 Factors
Many firms approach model risk tiering as analogous to credit analysis, creating a
score based on factors that roughly correspond to the two credit-risk dimensions of
probability of default and loss given default. In the case of model risk, the analogs are
the probability of model failure (how likely it is the model will experience a problem)
and the impact of model failure (the consequences for the firm if the model does
experience a problem). Some firms make this two-dimensional approach explicit,
although for many it is merely implicit in the selection of factors.
With regard to impact, firms generally consider materiality, which may depend
either on size (such as the notional value of exposures to which the model applies) or
on use. The example illustrated in Figure 3 includes a materiality factor. Many firms
consider both economic consequences and regulatory consequences; models that are
used for financial reporting or to meet regulatory requirements carry the potential for
greater impact, all else being equal. Figure 3 includes a regulatory exposure factor.
Some firms also add a separate factor to indicate the potential reputational impact
of model failure, particularly where this factor would raise some models to a higher
risk tier.7 The example in Figure 3 does not include a stand-alone reputational impact
factor.
As discussed above, the FHFA’s guidance (Federal Housing Finance Agency
2013) is the only formal supervisory reference to model risk tiering of which we
are aware.8 It highlights complexity, business impact and “extensiveness of use” as
factors to consider in a model risk-tiering approach. These factors align with the
approach described above. Complexity is a standard indicator of probability of model
7 Examples of such models include those made available for public or customer use, or models
used internally in investment advisory.
8 Office of the Superintendent of Financial Institutions (2017) includes a requirement for a “model
risk materiality classification scheme”, which may include “quantitative factors such as the size and
growth of the portfolio that the model covers in addition to its capital impact (eg, VaR). Qualitative
factors such as model age, complexity, purpose and strategic importance may also be considered,
where relevant, as long as this is done in a consistent fashion.”
failure, with more complex models being considered more prone to errors, ceteris
paribus. Figure 3 includes a model complexity factor. Business impact serves as a
blanket indicator of the potential impact of model errors. Extensiveness of use could
be another indicator of probability of model failure, as more extensive model use
may increase the likelihood of model misuse; or it could capture aspects of impact,
as more extensive model use would tend to mean that repercussions of any model
failure would be felt across a broader portion of the institution.
The example in Figure 3 includes an operational factor to cover the risk of model
failure due to lapses in the control environment within which the model resides.
While differences in the probability of model failure due to the nature of model con-
trols may be important, a factor of this type should only be included if the nature of
model controls is unlikely to be affected by tier assignments. Otherwise, the inclu-
sion of a controls factor in tiering, with tiering in turn influencing required controls,
may create instability in the risk-tier assignment due to potential circularity.
5.1.2 Components
Firms must identify one or more specific components for each factor to be reflected
in the model-tiering tool; these are the variables in the tree or scorecard. In choosing
components, builders of tiering tools generally show a preference for those that can
be objectively measured and can be transparently observed (preferably within the
existing inventory system). However, even firms that have already built robust model
inventories may find that they need to significantly expand the scope of model profile
elements collected in the inventory in order to capture the variables required by the
model risk-tiering tool.
In practice, a wide variety of components are used in model risk tiering. In this
section, we discuss examples of some of the typical components used in tiering tools.
5.1.2.2 Size. Firms use a variety of variables to indicate the size of the impact of
potential model failure on model results. Firms usually find estimating, or even clas-
sifying, the level of impact of a particular model on a given output to be surprisingly
challenging. The challenge arises when the organization attempts to estimate how
much a model error could affect the modeled result instead of the more tractable, but
arguably less relevant, calculation of the historical impact of the model on the result.
To gauge the impact of a model failure, firms generally use some combination of the
following:
impact of (or reliance on) the model on the final result (this is nearly always a
judgment).
The example in Figure 3 uses estimated dollar exposure (current book or market
value) as the only component of the materiality factor.
5.1.2.3 Use. Firms generally make categorical distinctions among model uses.
Examples of categories include general business, accounting and regulatory report-
ing, pricing and risk management, and fiduciary uses.
Most firms view general business use as presenting the least potential impact due
to model failure. Some common examples of model uses included in this category
are
employee compensation.
Many firms see financial accounting and regulatory reporting as presenting the
highest model risk. Some distinguish these as two different uses or even place spe-
cific regulatory uses in separate categories. An example of the latter approach is
evident in the sample MRS in Figure 3, where use in regulatory stress testing leads
to an override of the scoring results. Common examples of model uses included in
this category are
balance-sheet reporting,
The regulatory exposure factor in the example in Figure 3 has as its single
component the use of models in AML, financial or board reporting.
Some firms view models used for pricing and risk management as presenting risk
similar to that posed by models used for accounting and regulatory compliance,
while others view pricing and risk management uses as presenting less risk. Common
examples of pricing and risk management uses include
residential-mortgage pricing,
asset-liability management.
Firms show a wide range of practices in their treatment of models used to advise
clients or otherwise inform the allocation of client investments. Most consider the
use of models to carry out fiduciary duties as a significant contributor to model risk.
The case of AXA Rosenberg mentioned above, in which a financial firm that lost
most of its assets under management after paying US$217 million in restitution to
clients and a US$25 million fine to the SEC to settle a case in which it was found
to have concealed a model error, suggests the potential significance of such models.
However, some firms view the use of models for fiduciary activities as outside the
scope of their model governance frameworks.
5.1.2.4 Risk of model failure. Firms attempt to identify features of models, or of the
use of those models, that may make model failures more likely. Some may be fairly
subjective, such as whether the firm has prior experience with the modeled product
or modeling process, or model complexity (although some firms devise metrics to
characterize complexity). A sample of common components used to indicate risk of
failure is provided below:
number of users.
heuristic inputs can be very challenging to resolve. In part because of these chal-
lenges, many tiering-tool development processes revert to more easily observable
metrics, but this sometimes comes at the expense of useful information.
Different types of components require different treatment. Quantitative variables
with a wide range of potential values are generally converted into categorical vari-
ables represented by a number of integer values, such as categories one through five.
Many judgmental variables are also represented in practice by an integer scale; in
that case, narrative descriptions of the values on the scale may need to be developed
in discussions with stakeholders. Other components may be reduced to simple binary
measures. Regardless of which form a component takes in the tool, it is best to col-
lect the input data in the most granular form possible, to maximize its information
content and the flexibility of the tool-construction process.
Once the input data is collected, the usual data-hygiene processes must be com-
pleted, such as cleaning data and converting text entries to categorical variables. In
practice, this stage can be quite time-consuming, as respondents are likely to interpret
profile-element requests differently, regardless of the care taken in the development
of questionnaires or other instruments, and the descriptive information provided in
response may take a wide variety of styles. Enough time should be allowed for signif-
icant back-and-forth between the tool developer and respondents in order to develop
a data set that is as robust and consistent as it is practical.
combination of a model and each of its business uses as if they were unique models,
and tiering them on a stand-alone basis. As a result, a single model may receive
multiple risk-tier assignments depending on use. Firms using this approach often
base MRM requirements on the highest tier assigned to any model/use pair. More
commonly, however, firms apply their tiering tools to create a single tier for each
model, taking into account all of a model’s uses at once. Models with more model
uses are more likely to be assigned to higher risk tiers.
Which approach is more appropriate depends on the MRM framework and prac-
tices of the individual firm. Since the depth and frequency of validation activities are
frequently driven by a model’s risk-tier assignment, firms that conduct separate val-
idations for different model uses might reasonably choose to utilize the model/use
pair approach, while those that conduct validations covering all of a model’s uses
might use the model-centric approach.
Calibration is the process of tuning the algorithm that translates model data into
a risk-tier assignment. Effective calibration depends on a combination of factor
and component variable selection as well as properly setting weights and thresh-
olds associated with their use. Although this paper only addresses calibration for
scorecard-type tools, some calibration is required for decision trees as well.
No Yes
In firms without preexisting model risk-tier assignments, the first round of such
exercises frequently results in an excessive number of models being assigned to the
highest risk tier. Unless the model universe is such that very few models are utilized
in a large number of applications, such a distribution should not be accepted. As
discussed above, the MRS is a tool for distinguishing the riskiness of models within
an inventory; the placement of too many models into a single risk tier suggests that
it is not differentiating risk as intended. In that case, further calibration or changes to
the tool’s structure are likely necessary before the tool can be considered finalized.
For firms that have a legacy approach and therefore begin with an existing distri-
bution of models across risk tiers, the calibration of a new framework may proceed
somewhat differently, as illustrated by the flow on the right-hand side of Figure 4.
In such cases, maintaining the preexisting relative ranking of models is often seen
as a desirable feature for the new system, and risk managers may aim to minimize
the number of model risk assignments that change under the new tool. Assuming
the extent to which individual models are reused for different products or
in different applications (eg, a firm in which a small number of models are
utilized broadly will have a smaller inventory, with a larger proportion of
high-risk models, than one in which each use tends to generate its own model);
the comprehensiveness of the inventory, since models omitted from the inven-
tory are likely considered less crucial and would thus likely fill out the lower
risk tiers;
the alignment of tier-specific MRM requirements with the risks posed by the
models assigned to each tier; and
the proportion of the inventory that must be assigned to the top risk tier due to
supervisory pressure (eg, examiners may suggest that all models used in stress
testing should be assigned to tier 1).
Once a near-final calibration is established, the MRM group should lead a process
to resolve any issue with new risk-tier assignments through discussions with model
owners and business leads. At the end of this process, some firms will also choose
to set overrides in the MRS to guarantee a minimum tier assignment for models of
certain types or with certain uses, such as those required in regulatory-compliance
activities.
Our experience has been that initial model calibration frequently results in risk-
tier assignments for the model inventory with which managers are dissatisfied, par-
ticularly at firms with preexisting model risk-tier assignments. While a tiering tool
should not be required to perfectly replicate an inventory’s preexisting tiering, it is
important that the tool be able to generate tiers that are generally consistent with
expert opinion where that opinion is itself internally consistent. Depending on the
choice of inputs, calibrating an MRS to be broadly consistent with current risk-tier
assignments may prove easier than doing so with an MRT.
7 GOVERNANCE
Any enterprise risk management process should be subject to appropriate gover-
nance, and model risk tiering is no exception. Established governance practices
regarding model risk-tiering tools include the incorporation of risk-tiering proce-
dures into the model-governance framework and the assignment of clear roles and
responsibilities. A common practice is to give ownership of tool development and
maintenance to the MRM function. The head of the MRM function generally has
final sign-off on model risk-tier assignments, with model owners and model users
given an opportunity to challenge and escalate tier assignments in the event of dis-
putes. Most firms also include explicit or implicit requirements for reevaluation of
model risk-tier assignments on at least an annual basis, including a review of the
model profile elements used for model risk tiering.10
Best practices and other important aspects of model risk-tiering governance are
still emerging and not yet standardized. Leading firms maintain the model profile
elements necessary to populate risk-tiering tools in the model inventory system while
others maintain such information in spreadsheets or within a version of the tool itself.
Likewise, some firms require model owners to attest to the accuracy of the relevant
profile elements while others do not.
Supervisors appear to increasingly expect internal audits to evaluate the reason-
ableness of the model risk-tiering process. Although risk tiering is not directly
addressed in the US banking agency guidance, that guidance does state other super-
visory expectations for internal audits, including the assessment of the “accuracy
10 This reevaluation is generally part of the annual model review and meets the expectation in the
US banking agency guidance (2011 and 2017) that “[b]anks should conduct a periodic review –
at least annually but more frequently if warranted – of each model to determine . . . if the existing
validation activities are sufficient”.
and completeness of the model inventory” and the evaluation of “whether MRM is
comprehensive, rigorous, and effective”.
Another emerging practice in the governance of model risk tiering is the treatment
of some risk-tiering tools as models themselves. In firms where such tools are sub-
ject to the same documentation, validation, periodic review and even risk tiering as
the rest of the model inventory, this treatment is another step in establishing both
transparency and firm-wide acceptance of the model risk-tiering framework.
8 CONCLUSION
As models continue to play a growing and central role in business operations, MRM
will remain an area of focus for both senior executives and regulators. Leading finan-
cial institutions take a risk-sensitive approach to MRM in which model risk tiers drive
the frequency and depth of MRM activities. Model risk tiers are typically assigned
using a tool designed to ensure that they are determined consistently and objectively.
The design of model risk-tiering tools draws upon a variety of principles.
Model risk tiers should depend on a model’s inherent risk rather than a model’s
residual risk.
Model risk tiers provide an indication of the relative (not absolute) risk posed
by model use.
When designing a model risk-tiering tool, developers should consider the rel-
ative risk posed by models both within and across business units and legal
entities.
A model risk-tiering tool should broadly distribute the model inventory across
the risk tiers.
Model risk-tiering tools should produce results that are broadly in line with
management expectations.
In both literature and practice, we have observed two types of model risk-tiering
tools: decision trees and scorecards. Decision trees are designed such that users
answer a series of questions regarding a model to guide the assignment of a model’s
risk tier. Scorecards assign values based on a model’s attributes, with the values then
subjected to a weighting scheme to produce a model risk score; scores within defined
ranges are mapped to model risk tiers.
Calibration plays an important role in the tool-design process, determining both
the thresholds used to map model risk scores to tiers and the weights and thresholds
associated with individual model risk-tiering factors and components. Due to their
design, scorecard tools are generally easier to calibrate than decision trees.
Governance around the construction and use of model risk-tiering tools is an
evolving area. Similar to other aspects of risk management, effective model risk
tiering should rely on a clear allocation of roles and responsibilities with a com-
prehensive documentation framework to guide activities. Model developers, model
owners and users, model risk managers and internal auditors all play valuable roles in
developing, maintaining and ensuring compliance with effective model risk-tiering
procedures.
Model risk-tiering tools provide a systematic assessment of the relative risk posed
by different models subject to a given governance framework within a firm-wide
inventory. The clear value of these tools is that they provide a transparent and con-
sistent way to determine model risk-tier assignments, which are a critical building
block for the practice of consistent and effective risk-sensitive model governance.
DECLARATION OF INTEREST
The authors report no conflicts of interest. The authors alone are responsible for
the content and writing of the paper. The views and conclusions contained herein
are those of the authors and should not be interpreted as representing the views or
conclusions, either expressed or implied, of Promontory Financial Group.
REFERENCES
American Bankers Association (2017). Model risk management (“MRM”) survey. Report,
AMA. URL: www.aba.com/Groups/MRMWGdocs/SR1519survey1709.pdf.
Board of Governors of the Federal Reserve System (2011). Guidance on model risk
management. Supervision and Regulation Letter SR 11-7, Federal Reserve Board.
Committee of European Banking Supervisors (2006). Guidelines on the implementation,
validation and assessment of advanced measurement (AMA) and internal ratings based
(IRB) approaches. Guidelines, European Banking Authority.
Committee of Sponsoring Organizations of the Treadway Commission (2004). Enterprise
risk management: integrated framework. Executive Summary, September, COSO.
Deng, H., Runger, G., and Tuv, E. (2011). Bias of importance measures for multi-valued
attributes and solutions. In Proceedings of the 21st International Conference on Artificial
Neural Networks (ICANN 2011). Springer (https://doi.org/10.1007/978-3-642-21738-
8 38).
European Parliament and Council of the European Union (2009). Directive 2009/138/EC
of the European Parliament and of the Council of 25 November 2009 on the taking-up
and pursuit of the business of Insurance and Reinsurance (Solvency II). EUR-Lex.
European Parliament and Council of the European Union (2013). Regulation (EU) No
575/2013 of the European Parliament and of the Council of 26 June 2013 on prudential
requirements for credit institutions and investment firms and amending Regulation (EU)
No 648/2012 Text with EEA relevance. EUR-Lex.
Federal Deposit Insurance Corporation (2017). Adoption of supervisory guidance on
model risk management. Financial Institution Letter FIL-22-2017, FDIC.
Federal Housing Finance Agency (2013). Model risk management guidance. Advisory
Bulletin AB 2013-07, FHFA.
Keenan, S. C. (2015). Financial Institution Advantage and the Optimization of Information
Processing. Wiley (https://doi.org/10.1002/9781119053309).
Lewis, R. J. (2000). An introduction to classification and regression tree (CART) analy-
sis. Conference Paper, 2000 Annual Meeting of the Society for Academic Emergency
Medicine in San Francisco, California.
Mankotia, S., and Joshi, A. (2013). Measuring model risk: a practitioner’s approach. RMA
Journal, July 1.
Office of the Comptroller of the Currency (2000). Model validation. Bulletin 2000-16, OCC.
Office of the Comptroller of the Currency (2011). Supervisory guidance on model risk
management. Bulletin 2011-12, OCC.
Office of the Superintendent of Financial Institutions (2017). Enterprise-wide model risk
management for deposit-taking institutions. Guideline E-23, OSFI.
Securities and Exchange Commission (2014). Nationally recognized statistical rating
organizations. Final Rules, 17 CFR Parts 232, 240, 249 and 249b, SEC.
Securities and Exchange Commission (2016). Standards for covered clearing agencies.
Final Rule, 17 CFR Part 240, SEC.