Professional Documents
Culture Documents
Development and Validation of A Customized Competency
Development and Validation of A Customized Competency
www.emeraldinsight.com/1352-7606.htm
CCM
19,1 Development and validation of
a customized competency-based
questionnaire
90
Linking social, emotional, and cognitive
competencies to business unit profitability
Geoff Ryan
Competency International, London, UK
Lyle M. Spencer
Competency International, Saint Augustine, Florida, USA, and
Urs Bernhard
Competency International, Kerzers, Switzerland
Abstract
Purpose – The purpose of this paper is to report data empirically linking competencies of individual
leaders to business profitability and demonstrate that competencies are cross-culturally valid.
Design/methodology/approach – Participants in the initial competency study were 15 business
unit managers identified as high performing. Data were collected using Critical Incident Interviews
that were systematically coded using thematic analysis to identify the presence of competencies.
Competencies identified were then adapted into a behaviourally-based questionnaire used in a
follow-up validation study. Participants in the validation study (n ¼ 70) were managers from
North America and two European countries who were participants in a management development
program. Boss ratings of competencies were then correlated with business unit profitability.
Findings – A set of competencies was identified as predictive of unit profit growth in managers in
both North America and the European Union. Subsequent regression analysis showed that 17 per cent
of the variance in business unit profitability could be accounted for by four competencies, specifically
team leadership, developing others, achievement orientation, and impact and influence. Cross-cultural
validity was demonstrated to the degree that similar competencies predicted performance in both
North America and the European Union as evidenced by the correlation between boss rating of
subordinate competencies and profit growth.
Research limitations/implications – The initial study using Critical Incident Interviews was
conducted with a small sample size and did not employ a comparison group of average performers.
Practical implications – Initial competency research using empirical methods should be used to
help focus competency models used for selection, feedback, training, and performance management.
Originality/value – The study is one of the few published studies that link competencies to business
unit profitability. The paper demonstrates that competencies have a degree of cross-cultural validity.
Keywords North America, Europe, Competences, Profit, Surveys, Emotional intelligence,
Social intelligence, Critical incident interviewing, Competency modeling, Validity study,
Cross-cultural validity, European leaders
Paper type Research paper
Cross Cultural Management
Vol. 19 No. 1, 2012
pp. 90-103
q Emerald Group Publishing Limited The term “competency” has come to mean many things within the world of
1352-7606
DOI 10.1108/13527601211195646 human resource development (HRD) and has led to a proliferation of competing
definitions, applications, and claims related to the utility and validity of the construct. Competency-
This is also true in the emerging area of social and emotional intelligence which serves based
to extend traditional competency research with emerging areas of neuroscience. The
growth of competency-based human resource applications continues to show signs of questionnaire
growth around the world; however, the academic and applied research literature on
competencies in the workplace has trailed application (Boyatzis, 2008). Rapid changes
in the business environment demand flexible human resources strategies and 91
constructs which can be applied throughout complex, multi-national organizations. For
many organizations, competencies have attempted to address this pressing business
need. Changes brought on by globalization, technology, changing organizational
structures, and demographic shifts have made modern multi-national organizations
increasingly complex and interconnected. As the business world becomes more global
and interconnected, the pressure to develop and deploy reliable and valid methods for
assessing and developing competencies across national boundaries has become a
strategic priority for many organizations.
Legal and ethical concerns can also be seen as a potential driving force in the modern
competency movement. The standards for educational and psychological testing, which
were prepared by the American Educational Research Association, American
Psychological Association, and the National Council on Measurement in Education
(1999), require that measures be reliable and valid predictors of outcomes. US and
Canadian courts, under civil rights and pay equity laws, have ruled that “any (HR)
decision-making process [. . .] that affect an employee’s status in an organization are
tests and thus subject to scrutiny for adverse impact” (Latham and Wexley, 1981). The
effect of these rulings is to extend requirements for statistical reliability and validity to
any assessment for selection or promotion, any performance appraisal or any
development opportunity that may affect pay or career opportunities. However, in the
European Union the issue is less clear, but many observers believe that the trend started
in North American will likely become law within the European Union in coming years.
Aside from legal and ethical issues, significant controversy still remains regarding the
cross-cultural validity of competency frameworks: are they equally valid in diverse
cultures? For example, can models developed and validated in North America
be considered useful and valid in the European Union? In applied practice, the process of
actually empirically validating competency frameworks against outcome measures is
rarely seen; even less attention has been paid to empirically establishing the
cross-cultural validity of specific frameworks within multi-national organizations. While
such rigorous validations are often overlooked or viewed as an “academic” exercise which
is difficult to complete amidst the various competing priorities of senior executives and
senior HRD professions, such attitudes and beliefs often seriously undermine the validity
and practical utility of competency models. However, in our own applied research and
consulting (Ryan et al., 2009; Spencer, 2001; Spencer et al., 2008; Spencer and Spencer,
1993) we have found that taking the time to conduct such research yields much more than
merely “academic” benefits. Such applied research provides competency-based
applications additional predictive validity, credibility and allows HRD practitioners to
better quantify their contribution to their organizations (Spencer, 1986).
In this article we will review key findings from a multi-year applied research project
within a European-based firm which sought to better understand the specific
competencies which best predicted economic outcomes in their specific organization.
CCM The article describes the process of building and validating a competency-based
19,1 questionnaire for use in succession planning and management development within
North America and Europe. The article will document the organization’s efforts to
better understand, from an empirical standpoint, the contribution of competencies to
business unit profitability as well as efforts made to test the cross-cultural validity of
the initial model. This research was used within the organization to build a credible
92 business case in terms of economic value added and potential return on investment for
assessing and developing the competencies identified in the research as well as
validating the competency model to guide HR applications within the organization.
Defining competencies
With the popularity of competencies has come a myriad of competing definitions which
have been associated with the modern competency movement. In our own applied work,
we have found that the theory of competency originally put forward by McClelland
(1973) and then later refined by Boyatzis (1982) and Spencer and Spencer (1993) to be
especially well suited for research and applied practice. In this tradition, competencies
are defined as “underlying characteristics of the person that lead to or cause effective or
superior performance” (Boyatzis, 1982). To be appropriate for applied practice, these
characteristics must be able to be operationalized in a way that allows them to be reliably
measured so they can be empirically linked to criterion of interest to client organizations.
Research over the past 40 years has identified several hundred competencies, yet
careful scrutiny often finds a good deal of overlap in competency definitions. Research
has demonstrated that between 20 and 25 competencies typically account for most of
the variance in most jobs (Boyatzis, 1982, 2006; McClelland, 1973, 1998; Ryan et al.,
2009; Spencer et al., 2008; Spencer and Spencer, 1993; Spencer, 2001). While always
staying open to the possibility that new competencies may be discovered, this initial
framework serves as an excellent starting point for applied research.
For example, Table I shows a six-level scale for the competency of achievement
orientation. All indicators in the scale relate to the general definition given of the
competency achievement orientation. The different levels are descriptions of behaviour
Competency-
Achievement orientation: A concern for working well or for surpassing a standard of
excellence. The standard may be one’s own past performance based
(striving for improvement); an objective measure (results questionnaire
orientation); outperforming others (competitiveness); challenging
goals one has set; or even what has previously not been done before
(innovation)
1. Wants to do job well Works toward implicit standards of excellence. Tries to do job right 93
or well. May express frustration at waste or inefficiency
2. Works to meet Works to meet a standard set by management
others’standards
3. Creates own measures of Uses his or her own specific methods of measuring outcomes against
excellence a standard of excellence (not imposed by management)
4. Improves performance Makes specific changes in the system or in own work methods to
improve performance (e.g. does something better, faster, at lower
cost, more efficiently; improves quality, customer satisfaction,
morale, revenues), without setting any specific goal
5. Sets challenging goals Sets and acts to reach challenging goals for self or others.
“Challenging” means , 60 per cent chance of actually achieving the
goal – a definite stretch, but not unrealistic or impossible. Or cites
specific measures of baseline performance compared with better
performance at a later point in time
6. Makes cost-benefit analyses Makes decisions, sets priorities or chooses goals on the basis of
calculated inputs and outputs: makes explicit considerations of
potential profit, return on investment, or cost-benefit analysis
7. Takes calculated Commits significant resources and/or time (in the face of
entrepreneurial risks uncertainty) to increase benefits, improve performance, reach a
challenging goal, or innovate
8. Persists in entrepreneurial Takes numerous, sustained actions over time in the face of obstacles Table I.
efforts to reach entrepreneurial goal; or successfully completes Achievement
entrepreneurial endeavours orientation –
just-noticeable
Source: Adapted from Spencer and Spencer (1993) difference scale
Method
Organizational background
The client was a large multi-national electronic controls business headquartered
within the European Union with global offices and operations. The board had
identified that there was considerable variation in the profits generated by their several
hundred business unit managers worldwide. This was a matter of concern as
the business unit operations were essentially similar, regardless of location. Given this
variability in business unit profitability, a three-year study was commissioned by the
organization to gain a deeper understanding of how the competencies of business unit
managers contributed to business unit profitability. This would then allow the
organisation to better develop and target HR processes and programs to the specific
competencies most related to unit profitability. A competency study was planned so as
to develop a first version of a job competency model.
Owing to budget limitations as well as time constraints a hybrid approach was
developed. Rather than following the normal method of comparing a sample of
superior performing business unit managers with a sample of average performers a
two-stage approach was utilized.
In the initial stage critical incident interviews were conducted with a sample of Competency-
superior performers to develop content for a competency questionnaire. In the second based
stage responses to the questionnaire items were used to conduct t-tests between the
responses related to superior performers to those of average performers so as to questionnaire
indentify the differentiating competencies.
Results
Competency model development. Competencies were then ranked by value and the top
12 selected to form a competency model. These results were reviewed by the study
participants in a two-day workshop and final modifications made to the model. The
findings of the analysis of the competency behaviours coded from the critical incident
interviews revealed 12 competencies which accounted for 83.5 per cent of the total
number of behavioural indicators displayed by the 15 study participants.
CCM These 12 competencies were ranked by value, i.e. average frequency multiplied by
19,1 average level to form the competency model as shown in Figure 1.
Task analysis. In addition to coding competencies from the critical incident interview
transcripts, the task being worked on when specific competencies were demonstrated
was also noted. In total, 46 different tasks were identified from the interview data and
were then categorised into nine broad task groups on the basis of task similarity. This
96 allowed a task group/competency matrix to be developed, which had the task groups on
one axis and the competencies, as coded from the interviews, on the other. The cells of the
matrix were populated with the total frequencies of the individual competencies that
appeared in relation to each task group from 15 business unit managers that were
interviewed in the initial study. The competencies themselves were clustered into six
general themes; managing tasks and process, relating to others, leading others,
managing others, thinking and solving problems, and managing self. The clustering
strategy closely follows the original competency clusters as outlined in Competence at
Work (Spencer and Spencer, 1993). For the purpose of the task analysis all 20 original
competencies are included for the purpose of illustration, although the actual competency
model implemented within the client organization only included the 12 “highest value”
competencies. The task group/competency matrix is shown in Table II.
As can be seen in the table some of the more frequently demonstrated competencies
included achievement orientation (69), impact and influence (71), team leadership (58),
interpersonal understanding (50), analytical thinking (50), teamwork and
collaboration (46), and initiative (45), all of which were included in the competency
Figure 1.
Emotional, social and
cognitive competencies
ranked by value
Competency-
Task group
Competency cluster Competency T1 T2 T3 T4 T5 T6 T7 T8 T9 Total based
Managing tasks and Achievement 1 2 18 5 6 2 29 4 2 69
questionnaire
process orientation
Concern for order 1 1 7 4 4 3 9 3 1 33
Initiative 0 2 11 4 8 1 14 1 4 45 97
Information seeking 2 1 11 4 4 3 11 1 2 39
Relating to others Interpersonal 1 1 17 8 9 2 8 2 2 50
understanding
Customer service 0 0 11 1 1 4 9 1 5 32
orientation
Leading others Impact and influence 1 3 23 5 4 10 20 1 4 71
Organizational 0 1 9 2 1 2 7 0 0 22
awareness
Relationship building 0 0 0 0 1 2 2 0 2 7
Managing others Developing others 0 0 14 6 8 0 5 4 0 37
Directiveness 1 1 13 4 4 1 1 3 0 28
Teamwork and 1 3 21 4 8 1 7 1 0 46
cooperation
Thinking and solving Team leadership 0 3 35 2 9 1 8 0 0 58
problems Analytical thinking 0 8 10 6 7 3 9 3 4 50
Conceptual thinking 4 2 14 7 2 0 7 1 0 37
Use of expertise 0 1 1 0 2 0 0 0 1 5
Managing self Self-control 0 0 1 0 0 0 0 0 0 1
Self-confidence 1 2 11 2 3 2 6 0 1 28
Flexibility 0 0 2 0 2 2 5 0 1 12
Organizational 0 0 6 0 1 0 2 0 0 9
commitment
Total indicators 13 31 235 64 84 39 159 25 29
Notes: T1, monitoring plans and performance; T2, developing plans and strategies; T3, Table II.
managing change; T4, dealing with problems; T5, developing, coaching and team building; T6, Task group/competency
communicating; T7, selling-related activities; T8, quality control; T9, managing environment matrix
model implemented within the organization. What also stands out in the table is the
frequency with which business unit managers were engaged in tasks related to
managing change (234), selling-related services (159), and developing, coaching and
team building (84). This understanding of how frequently specific competencies were
demonstrated while business unit managers went about specific job-related tasks
provided the client organization with additional insight into how the organizations best
performers went about their job.
Results
The results of the screening are shown in Table IV. As can be seen seven of the
12 competencies (achievement orientation, impact and influence, developing others,
teamwork and cooperation, analytical thinking, concern for quality and order, and
conceptual thinking) had correlations of R 2 of 0.05 and above with the continuous
standardized score for profit growth measure. For the profit growth measure expressed
as a percentile rank, information seeking also correlated at R 2 0.05 in addition to the
seven competencies listed above. Thus, there was little difference between the strength
of the relationship with the profit growth measure expressed as a continuous
standardized score or when shown as a percentile rank with outliers removed.
The t-tests showed that eight competencies (achievement orientation, impact and
influence, developing others, initiative, interpersonal understanding, teamwork
Discussion
The main focus of the research reported here was to establish the predictive validity of the
initial competency model established using operant data gathered from critical incident
interviews. A secondary objective was to see if competencies measured using a
respondent measure (i.e. questionnaire) could predict performance in a North American
sample as well as in two separate countries in the European Union. The research
documented here serves as a limited validation of both points. The research also
demonstrates that empirical competency research methods which identify reliably
measurable competencies which statistically predict business economic results have
significant utility to multi-national organizations. Such research gives HR applications
(e.g. staffing, training, performance appraisal, succession planning, selection, and
compensation) based on competency models enhanced credibility with organizational
Change statistics
Adjusted SE of the R2 F Sig. F
2
Model R R R2 estimate change change df1 df2 change
References
American Educational Research Association, American Psychological Association, National
Council On Measurement in Education (1999), Standards For Educational and
Psychological Testing, American Psychological Association, Washington, DC.
Boutet, M., Milsom, J. and Mercer, C. (2000), “Revising management competencies: ensuring
cross-cultural validity”, Competency and Emotional Intelligence, Vol. 7 No. 2, pp. 12-26.
Boyatzis, R.E. (1982), The Competent Manager: A Model of Effective Performance, Wiley,
New York, NY.
Boyatzis, R.E. (1998), Transforming Qualitative Information, Sage, Thousand Oaks, CA.
Boyatzis, R.E. (2006), “Using tipping points of emotional intelligence and cognitive competencies
to predict financial performance of leaders”, Psicothema, Vol. 18, pp. 124-31.
Boyatzis, R.E. (2008), “Competencies in the 21st century”, Journal of Management Development,
Vol. 27 No. 1, pp. 5-12.
Chong, E. (2008), “Managerial competency assessment: a cross-cultural study of American and Competency-
East Asian managers”, Journal of Business Research, Vol. 61, pp. 191-200.
Latham, G.E. and Wexley, K.N. (1981), Increasing Productivity Through Performance Appraisal,
based
Addison-Wesley, Reading, MA. questionnaire
McClelland, D.C. (1961), The Achieving Society, The Free Press, New York, NY.
McClelland, D.C. (1973), “Testing for competency rather than intelligence”, American
Psychologist, Vol. 28 No. 1, pp. 1-40. 103
McClelland, D.C. (1998), “Identifying competencies with behavioral event interviews”,
Psychological Science, Vol. 9 No. 5, pp. 331-40.
Ryan, G., Emmerling, R.J. and Spencer, L.M. (2009), “Distinguishing high performing European
executives: the role of emotional, social and cognitive competencies”, Journal of
Management Development, Vol. 28 No. 9, pp. 859-75.
Spencer, L.M. (1986), Calculating Human Resource Costs and Benefits, Wiley, New York, NY.
Spencer, L.M. (2001), “The economic value of emotional intelligence competencies and EIC-based
HR programs”, in Cherniss, C. and Goleman, D. (Eds), The Emotionally Intelligent
Workplace: How to Select for, Measure, and Improve Emotional Intelligence in Individuals,
Groups, and Organizations, Jossey-Bass, San Francisco, CA, pp. 45-82.
Spencer, L.M. and Spencer, S. (1993), Competence at Work: Models for Superior Performance,
Wiley, New York, NY.
Spencer, L.M., Ryan, G. and Bernhard, U. (2008), “Cross-cultural competencies in a major
multinational industrial firm”, in Emmerling, R.J., Shanwal, V. and Mandal, M. (Eds),
Emotional Intelligence: Theoretical and Cultural Perspectives, Nova Science, New York, NY,
pp. 191-208.
Corresponding author
Geoff Ryan can be contacted at: [email protected]