See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/349854207

Sustainable Investing: The Black Box of Environmental, Social, and Governance

(ESG) Ratings

Article  in  Journal of Wealth Management · March 2021

DOI: 10.3905/jwm.2021.1.130

CITATIONS READS
18 1,796

2 authors, including:

Subhash Abhayawansa
Swinburne University of Technology
51 PUBLICATIONS   1,266 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Investment decision making View project

"Extending Intellectual Capital Through Integrated Reporting" Special issue call for papers from Journal of Intellectual Capital View project

All content following this page was uploaded by Subhash Abhayawansa on 16 March 2021.

The user has requested enhancement of the downloaded file.

Paper Title: Sustainable investing: The black box of environmental, social and governance (ESG)
ratings

Subhash Abhayawansa, Associate Professor in Accounting and Finance, Swinburne University of

Technology, Hawthorn, Australia (email: [email protected])

Shailesh Tyagi, Founding & Managing Partner - Asia Pacific, Viridi Global.
([email protected]).

Accepted for publication in Journal of Wealth Management

1
Paper Title: Sustainable investing: The black box of environmental, social and governance (ESG)
ratings

Abstract
Environmental, social and governance (ESG) investing is becoming mainstream, and the COVID-
19 pandemic has amplified the momentum. The interest in ESG investing has created greater
demand for ESG data, ratings and rankings together with a proliferation of agencies offering these
products which are unquestioningly relied on by investors, academics and regulators. Research
highlights that different ESG ratings and rankings produce significantly different assessments of
the ESG performance of companies. In this paper, we examine the causes of the differences in the
ratings and ranking produced by different agencies. It is found that the divergences between raters
can be attributed to differences in defining ESG constructs (i.e., theorisation problem) and
methodological differences (i.e., commensurability problem). While users of ESG ratings and
rankings are advised to study the definitions and methodologies prior to their use, lack of
transparency about the data sources, weightings and methodologies makes it difficult to ensure that
companies’ true ESG performance is accounted for when making portfolio selection and
investment decisions. As a solution, we suggest that instead of attempting to compare and contrast
ratings and rankings of different agencies, investors should determine ESG constructs material to
their investment strategy and match them with an ESG ratings/rankings product that closely
resemble those constructs.
Highlights:
1. There are significant divergences between the ratings and rankings provided by different
ESG rating agencies.
2. Differences between various ESG ratings and rankings are caused by the differences in
defining ESG constructs (i.e., theorisation problem) and differences in methods applied for
measuring ESG performance of companies (i.e., commensurability problem).
3. Agencies providing ESG rating and ranking are not transparent about what constitute ESG
performance and how ESG performance is measured, including information sources used.
4. The theorisation, commensurability and transparency problems contribute to masking the
true ESG risks and performance of companies

Keywords: Environmental, social and governance (ESG) ratings, socially responsible investing,
commensurability, sustainability reporting
JEL Code: P45

2
SUSTAINABLE INVESTING: THE BLACK BOX OF ENVIRONMENTAL, SOCIAL, AND
GOVERNANCE (ESG) RATINGS

Overview

Sustainable investing is an approach to investing that considers environmental, social and

governance (ESG) factors in portfolio construction and management. Once the domain of socially
responsible investors trying to exclude sustainability laggards from their portfolios, sustainability
investing is no longer a niche approach. The assets under management using sustainable investing
strategies1 surpassed US$ 30.7 trillion (or around 9% of global debt and equity) in the five major
markets of Europe, the United States, Canada, Japan, and Australia and New Zealand at the
beginning of 2018 (Global Sustainable Investment Alliance, 2018). In the two years to 2020,
professionally managed funds adopting sustainable investing strategies in the U.S alone increased
by 42% to US$17.1 trillion (US SIF, 2020). The inflows to ESG investing funds have continued to
grow in 2020 with the full-year amount for 2019 being achieved by the middle of the year2 (Hale,
2020). A 2019 survey of more than 550 members of the CFA Institute spread out globally shows
that more than half of the quantitative analysts or portfolio managers incorporate environmental
and social factors and two-thirds incorporate governance factors into their investment analysis
(Singh and Peters, 2019).
The rise in sustainable investing has given birth to a new industry that trades in ESG data
and ratings on companies, funds dedicated to rated companies and ESG index providers. Hawley
(2017) notes that there are more than 600 products from over 150 organisations providing ESG
data, ratings and rankings, with MSCI Inc and Sustainalytics being the leading players. The
industry is dynamic with new ratings appearing and disappearing, and rating organisations merging
and realigning at a rapid pace (Sadowski et al., 2010a). ESG ratings are used by academics, for
instance, to understand how the financial and market performance of sustainable companies differs
from others and what drives those differences. ESG ratings as well as academic research based on
them can potentially influence regulators, impacting policies and regulations relating to the
disclosure and use of ESG data3. The unquestioning reliance by various parties, including
investors, academics and regulators, on ESG ratings makes it important to understand whether ESG

1
Sustainable assets under management include assets that are managed using one of the following seven strategies: (1)
negative/exclusionary screening; (2) positive/best-in-class screening; (3) norm-based screening; (4) ESG integration;
(5) Sustainability themed investing; (6) impact/community investing; and (7) corporate engagement and shareholder
action (see, for more details, Global Sustainable Investment Alliance, 2018).
2
Flows are estimated for 315 sustainable open-end and exchange-traded funds available to U.S. investors which
include equity, fixed-income, allocation, and alternatives funds that have an ESG, impact, or sustainable sector focus
(see Hale, 2020).
3
For example, the U.S. Department of Labor (2020) new rule relating to the consideration of Financial Factors in
Selecting Plan Investments prohibits Employee Retirement Income Security Act (ERISA) plan fiduciaries from
choosing investments to promote environmental, social, and other public policy goals unrelated to the interests of plan
participants and beneficiaries. Also, recommendations of the SEC Investor Advisory Committee (2020) relating to
ESG disclosure show that the growth in ESG data, ratings and ranking providers has been taken into consideration in
the policy discussion relating regulating ESG reporting.
3
ratings of companies reflect true ESG performance of those companies and whether different ESG
ratings converge. The purpose of this paper is to identify the causes of the differences in the ratings
and ranking produced by different agencies.

All ESG ratings are not made equal

Several studies and media articles point out that ESG ratings produced by different agencies
show significant divergence (e.g., Chatterji et al., 2016; Yont et al., 2018). Allen (2018) shows that
in September 2018, FTSE placed Tesla last among the global automotive companies on ESG
performance while MSCI placed it at the top and Sustainalytics' placed it somewhere in the middle.
Semenova and Hassel (2015) explored the convergent validity of the environmental ratings of
MSCI (formerly known as Kinder, Lydenberg, and Domini Research & Analytics; KLD),
Thomson Reuters’ ASSET4 (Now Refinitiv) and Global Engagement Services and found that
while the ratings have common dimensions, on aggregate, they do not converge. Dorfleitner et al.
(2015) compared individual environmental, social, governance and economic scores as well as the
aggregate ESG scores of three rating products: Thomson Reuters’ ASSET4, MSCI/KLD ratings
and ESG data set of Bloomberg. Using a subsample containing companies covered by all three
rating providers between 2002 and 2012 they show that correlations between ASSET4 and
Bloomberg were as high as 0.62 for the aggregate score and varied between 0.47 and 0.60 for
individual dimensions, whereas MSCI/KLD ratings showed little resemblance to the other two
ratings (correlation varying between 0.05 and 0.39). Similarly, Chatterji et al. (2016) conclude that
the six ratings they compared (i.e., MSCI/KLD, ASSET4, Calvert, FTSE4Good, DJSI, and
Innovest) exhibit low convergence in their assessments of ESG factors. Futher, Hawley (2017)
showed that the correlation between the rankings based on the MSCI/KLD and Fortune Magazine’s
‘Best 100 Firms’ is merely 14%.

Why do various ESG ratings differ?

Our review of the literature shows that definitional and methodological differences
contribute to the low convergence between different ESG ratings. Disagreements about the
definitions of ESG factors, their composition and the weightings for each of the E, S ad G factors
are known as the theorisation problem, and having a variety of approaches or methodologies to
measure the ESG factors (even if definitions are agreed upon) is called the commensurability
problem (Hawley, 2017). The theorisation problem originates in the different views held by rating
agencies on the types of ESG factors considered financially material and the degree to which they
are considered material (i.e., the differences in their materiality maps). Dorfleitner et al. (2015,
p.256) identify the theorisation problem in their comparison of the compositions of ASSET4,
MSCI/KLD and Bloomberg ESG ratings. As an example of the theorisation problem, they
highlight that “animal testing is solely covered by ASSET4, whereas the question of whether a
company’s practice complies with environmental regulations is only considered by Bloomberg and
KLD”. Chatterji et al. (2016), upon comparing social ratings from six ESG raters find differences
between ratings even when adjusted for the explicit differences in the definitions of ESG factors.

4
Thus, Chatterji et al. (2016) demonstrate that the commensurability problem is independent of the
theorisation problem.
The commensurability problem relates to the differences in how the individual ESG
dimensions are measured, especially relating to the level of detail. One aspect of the
commensurability problem relates to the differences in the way financially material ESG factors
are measured using indicators. Dorfleitner et al. (2015) explain that the three ratings they examined
broadly cover the same aspects within the social dimension, and indicators within it account for
approximately half of the total data points/binary indicators. However, they note that the number of
indicators dedicated to particular aspects within the social dimension, such as health and safety,
differed significantly between the three ratings. The extent to which different raters conflate ESG
impacts (which indicate ESG risks) and ESG performance contributes to low commensurability
(Mattingly and Berman, 2006), and constitutes the other aspect of the commensurability problem.
Commenting on the environmental dimension of ESG, which was the subject of their study,
Mattingly and Berman (2006, p.250) claim that “industry-specific risk drives EP [environmental
performance] and that performance and risk are different constructs to be clearly separated”.
Notwithstanding, Dorfleitner et al. (2015) found that ESG risk measures have the lowest level of
correlation between different raters.
Finally, the commensurability problem also relates to the differences in the information
sources used in developing the ratings. The information sources vary from companies’
sustainability reports and other publicly disclosed information to surveys and interviews with the
company and independent information channels. Sadowski et al. (2010b) state that about one-half
of the 120 raters they examined rely only on public information sources while the rest use either
corporate self-disclosure alone or self-disclosure combined with public information sources. The
use of information disclosed publicly by companies can bias the ratings in favour of larger
companies (as larger companies provide more disclosures than smaller companies after controlling
for their level of corporate social performance) and companies domiciled in geographical regions
such as Europe where there are greater regulations and relatively mature traditions of sustainability
reporting. Similarly, Sadowski et al. (2010b) identify a bias towards companies responding to
information requests.
Moreover, the ratings of those agencies relying on publicly disclosed information, which is
often based on annual reporting cycles, may not reflect the same period as that of the agencies
using more timely information sources. Then, there is the problem of reliability and comparability
of information disclosed in sustainability reports, as there are neither commonly accepted standards
for sustainability reporting nor a requirement to have the reports audited. Also, depending on the
sustainability reporting guidelines adopted, companies might either disclose ESG information with
the investors in mind (e.g., if Sustainability Accounting Standards Board guidelines are adopted) or
take in to account a multiplicity of stakeholders (e.g., if Global Reporting Initiative Standards are
applied). Hence, the same data source can provide different perspectives about different
companies, making comparisons of ESG performance and risks across companies difficult.

5
Differences between ESG ratings and actual ESG performance

One can identify several limitations in the generally adopted rating methodologies that
contribute to a gap between the perceived (based on ESG ratings) and true ESG performance and
risks of companies. As the scores on various indicators are added together to generate an overall
ESG rating and ratings on each of the three factors (i.e., E, S and G), trade-offs between indicators
are inevitable (Hawley, 2017). For instance, a company that scores exceptionally on the diversity
aspect might, nonetheless, perform poorly on board independence for having directors with longer
tenures, smaller boards and a smaller share of independent directors (Yont et al., 2018). Most
rating methodologies are likely to assign these two indicators similar weightings and combine
them, although low board independence might constitute a greater risk than the absence of a
gender-diverse board. Combining metrics masks the important differences between companies in
relation to performance and risk characteristics by aggregating empirically and conceptually
distinct ESG constructs (Mattingly and Berman, 2006).
Similarly, most raters fail to distinguish between disclosure (e.g., carbon emissions) and
performance (e.g., actions to lower emissions) and disregard the need to weigh them differently
(Hawley, 2017). These problems are aggravated when multiple aspects relating to a particular ESG
dimension are combined in one indicator (in contrast to one dimension) by one rater while they are
kept separate by another (Dorfleitner et al., 2015). Moreover, raters are criticised for not
adequately focusing on material issues and trying to compare companies across industries which
differ on material ESG issues (Sadowski et al., 2010a). Related to determining materiality is the
time period over which given aspects of E, S or G factors are considered material. For example,
environmental factors such as loss of biodiversity or stranded assets resulting from climate change
regulation might only be material in the long-term, whereas other transitional risks of climate
change or release of toxic waste might be material in the short term, despite the time horizons over
which the materiality of a given E, S, or G aspect differing between industries. Rating agencies do
not often disclose the time-horizons used for determining materiality, thus, denying users of critical
information to understand companies’ true ESG risks and ESG performance. The result of these
limitations is ESG ratings that are incapable of representing organisational realities. Allen (2018)
argues that “Just because you can measure this stuff doesn't mean that you necessarily should.”

Exercise caution

While the quality and comparability of ESG data and ratings remain hotly contested,
investors are cautioned against over-relying on, even, top-level ESG scores (Chatterji et al., 2016;
Yont et al., 2018). At the very least, the users of ESG data and ratings should understand what the
methodology of the raters they choose actually measures. Unfortunately, this too is problematic as
most providers of ESG ratings and formulators of ESG rankings are not transparent, or only
transparent to a degree about their data sources, weighting and methodologies (Hawley, 2017). For
instance, some raters disclose categories and weightings, but not many explain how these
categories and weightings were determined or the specific criteria applied within the categories

6
(Sadowski et al., 2010a). The transparency problem is unlikely to go away as the rating
methodologies (e.g., ESG indicators and weightings), data sources (e.g., interview participants) and
measurement instruments (e.g., questionnaires) are proprietary and considered intellectual property
of the raters. The usefulness of ESG ratings is further complicated by the potential conflicts of
interest of some rating and ranking agencies due to having connections with the companies they
rate or produce indices based on their ratings.

Concluding remarks

The use of ESG ratings is tipped to increase with the market starting to price company
specific ESG risks and integrate ESG considerations alongside traditional financial analysis.
Together with this growth, regulations on ESG disclosure, ESG ratings and rankings and use of
ESG factors in the investment processes are likely to follow (see, for example, Commission, 2020;
U.S. Department of Labor, 2020). The theorisation, commensurability and transparency problems
associated with ESG ratings are unlikely to disappear anytime soon. Thus, the only safeguard for
investors is educating themselves on the types and amount of ESG data, ratings and rankings
available, and the strengths and limitations of various methodologies and nuanced processes used
by the raters (LaBella et al., 2019). We do not claim that ESG ratings are unusable: to the contrary,
they can be valuable if used with caution and as one of the inputs. The best way to embed ESG
considerations within the investment process is by first determining what ESG constructs are
material for the investor and then identifying an ESG rating that includes measures that closely
resemble those constructs.

7
 References

Allen, K. (2018), "Lies, damned lies and ESG rating methodologies", Financial Times.
Chatterji, A. K., Durand, R., Levine, D. I. and Touboul, S. (2016), "Do ratings of firms converge?
Implications for managers, investors and strategy researchers", Strategic Management
Journal, Vol. 37 No. 8, pp. 1597-1614.
Commission, S. a. E. (2020), Modernization of Regulation S-K Items 101, 103, and 105 (U.S.
Securities and Exchange Commission: Washington D.C).
Dorfleitner, G., Halbritter, G. and Nguyen, M. (2015), "Measuring the level and risk of corporate
responsibility – An empirical comparison of different ESG rating approaches", Journal of
Asset Management, Vol. 16 No. 7, pp. 450-466.
Global Sustainable Investment Alliance (2018), 2018 Global Sustainable Investment Review
(Global Sustainable Investment Alliance).
Hale, J. (2020), "Sustainable Funds Continue to Rake in Assets During the Second Quarter",
available at: https://www.morningstar.com/articles/994219/sustainable-funds-continue-to-
rake-in-assets-during-the-second-quarter (accessed 2020 December 12).
Hawley, J. (2017), ESG Ratings and Rankings: All over the Map. What Does it Mean? (Truevalue
Labs).
LaBella, M. J., Sullivan, L., Russell, J. and Novikov, D. (2019), "The Devil is in the Details: The
Divergence in ESG Data and Implications for Responsible Investing", QS Investors.
Mattingly, J. E. and Berman, S. L. (2006), "Measurement of Corporate Social Action: Discovering
Taxonomy in the Kinder Lydenburg Domini Ratings Data", Business & Society, Vol. 45
No. 1, pp. 20-46.
Sadowski, M., Whitaker, K. and Buckingham, F. (2010a), Rate the Raters: Phase One - Look Back
and Current State (SustainAbility: New York).
Sadowski, M., Whitaker, K. and Buckingham, F. (2010b), Rate the Raters: Phase Two - Taking
Inventory of the Ratings Universe (SustainAbility: New York).
SEC Investor Advisory Committee (2020), Recommendation of the SEC Investor Advisory
Committee Relating to ESG Disclosure (U.S. Securities and Exchange Commission:
Washington D.C).
Semenova, N. and Hassel, L. G. (2015), "On the Validity of Environmental Performance Metrics",
Journal of Business Ethics, Vol. 132 No. 2, pp. 249-258.
Singh, m. and Peters, S. (2019), The Case for Quarterly and Environmental, Social, and
Governance Reporting (CFA Institute: New York).
U.S. Department of Labor (2020), Financial Factors in Selecting Plan Investments (US
Department of Labour: Washington D.C).
US SIF (2020), 2020 Report On US Sustainable, Responsible And Impact Investing Trends (US
SIF The Forum for Sustainable and Responsible Investment: Washington D.C).
Yont, C., Jamie, A. and Zhou, M. (2018), Hard decisions: Asia faces tough choices in CG reform
(CLSA and Asian Corporate Governance Association Hong Kong).

View publication stats