Professional Documents
Culture Documents
Furr (2011) CAP 2 Scale Construction and Psychometrics For Social and Personality Psychology.
Furr (2011) CAP 2 Scale Construction and Psychometrics For Social and Personality Psychology.
and Psychometrics
for Social and
Personality
Psychology
R. Michael Furr
This chapter presents principles and practices that are among the broadest and
most fundamental issues for scale construction, modification, use, evaluation, and
interpretation. The points are rather straightforward but are vitally important in
conducting informative research. Thus, this chapter provides nontechnical over-
views of each point, to be complemented by greater exploration and depth later in
this volume.
Several of these principles and practices strike me, as an editor, reviewer, and
reader of social/personality research, as being somewhat under-appreciated. To
retain focus on those issues, this chapter bypasses issues that, though fundamental
to scale construction and psychometrics, seem generally well-known and well-
implemented. Indeed, much social/personality research is based upon measurement
that is well-conceived and appropriately-executed. This discussion is intended to
raise awareness and understanding of issues that, if appreciated even more widely,
will enhance the generally good conduct and interpretation of research. The issues
are summarized in Table 2.1.
Most facets of the process and principles covered in this chapter apply to
all forms of psychological measurement. For example, this chapter addresses
the need to articulate the construct and context of a measurement strategy, the
need to evaluate psychometric properties, and the need to revise the measure-
ment strategy if necessary—all of which apply to measurement strategies
such as “tests” of maximal performance, reaction time, behavioral obser-
vations, physiological, measures, choices and decisions, informant-reports,
and so on.
In addition, this chapter outlines scale construction in terms of four steps
(Figure 2.1). Reflecting contemporary social/personality psychology (John &
Benet-Martinez, 2000), this chapter (and this volume more generally) blends sev-
eral approaches to scale construction. It involves rationally-focussed item-writing,
attention to scale dimensionality and internal coherence, and empirical examination
of the scale’s psychological meaning.
Table 2.1 Under-appreciated principles and practices in scale construction, use, evaluation, and
interpretation
Principles and practices
1 Examination and interpretation of reliability and validity
a) Psychometric properties and quality of the current data should be evaluated
b) Psychometric properties and quality of the current data should be considered when
using scales and when drawing psychological implications from observed effects
2 Dimensionality (i.e., factor structure)
a) Dimensionality should be evaluated and considered in scale construction, use, and
evaluation
b) Coefficient alpha should is not an index of unidimensionality, the “eigenvalue greater
than one” rule should be avoided, and oblique rotations are preferable to orthogonal
3 Ad-hoc scales
a) Previously-validated scales are preferable to ad hoc scales
b) Ad hoc scales should be examined psychometrically, requiring more than assumed face
validity
4 Modified scales
a) Previously-validated original scales are preferable to modified scales
b) Modified scales should be examined psychometrically, not assumed to have the same
dimensionality, reliability, and validity of original scales
5 Brief/single-item scales
a) Sufficiently-long scales are preferable to overly-brief scales
b) Brief/single-item scales should be examined psychometrically, and their psychometric
quality and potential limitations should be appropriately discussed
6 The use of scales across groups
a) If scales are used in groups that might differ psychologically from those within which the
scales were developed and validated, then the psychometric properties/differences
should be examined, understood, and potentially rectified.
b) Careful translation does not guarantee psychometric stability across linguistically-
differing groups
7 Difference scores (i.e., gain scores, change scores, discrepancy scores)
a) Alternatives to difference scores should be considered strongly.
b) Difference scores should be used with attention to their components and their
psychometric quality.
8 Advanced psychometric perspectives (e.g., Confirmatory Factor Analysis, Generalizability
Theory, and Item Response Theory)
a) Advanced perspectives offer some important differences/advantages, as compared to
traditional psychometric theory; thus, they might be highly-appropriate methods of
understanding and evaluating the psychometric properties of a given scale
b) Producers and consumers of research should be prepared to provide and/or interpret
information obtained from these perspectives when necessary
Final
scale
Dimensionality
A scale’s dimensionality, or factor structure, reflects the number and nature of
variables assessed by its items. Some questionnaires, tests, and inventories are
unidimensional, with items cohering and reflecting a single psychological varia-
ble. Other questionnaires are multidimensional, with sets of items reflecting
different psychological variables.
Usually based upon factor analysis, an accurate understanding of the number
and nature of a scale’s dimensionality directly affects its scoring, psychometric
quality, and psychological meaning. Dimensionality dictates the number of mean-
ingful scores that a scale produces for each participant. If a scale includes two
independent dimensions, its items should be scored to reflect those dimensions.
For example, the Positive Affect Negative Affect Schedule (PANAS; Watson
et al., 1988) is a multidimensional questionnaire that produces one score for
Positive Affect (PA) and another for Negative Affect (NA). Researchers should
not combine items across the two dimensions, as it would produce scores reflect-
ing no coherent psychological variable. By dictating the number of meaningful
scores derived from a questionnaire, dimensionality also directs researchers’
evaluations of reliability and validity. That is, researchers must understand the
psychometric quality of each score obtained from a questionnaire. For example,
the PANAS has been developed and used with psychometric attention to each of
its two “subscales.” Thus, researchers who develop and use psychological scales
must understand the dimensionality of those scales. This is true even for short
scales that might appear to reflect a single psychological variable. Inaccurate
understanding of dimensionality can produce scores that are empirically and
psychologically meaningless.
An important related point is that a scale’s dimensionality is not clearly
reflected by the familiar Cronbach’s coefficient alpha. Based upon a scale’s internal
Ad Hoc Scales
Occasionally, researchers create scales to measure specific constructs for a study.
Of course, scale-development is important for psychological research, and there
are good reasons to create new scales (e.g., one might wish to measure a construct
for which a scale has not yet been developed). However, there are two caveats
related to ad hoc scales.
The first caveat is that previously-validated scales are generally preferable to
ad hoc scales. That is, when well-validated scales exist for a given construct,
researchers should strongly consider using those scales rather than a new scale.
For example, there are many well-validated self-esteem scales differing in length,
psychological breadth, and dimensionality. With such diversity and known psy-
chometric quality within easy reach, there seems little reason to assemble a new
ad hoc self-esteem scale.
The second caveat is that, if ad hoc scales are created, they require psychomet-
ric evaluation, including validity evidence that goes beyond face validity. Ideally,
scales that are intended to measure important psychological variables are devel-
oped through a rigorous process emphasizing psychometric quality. However,
ad hoc scales sometimes seem to be applied without much evaluation, with
researchers apparently relying upon on face validity and assuming that items
obviously reflect the intended construct. Not only should researchers examine the
dimensionality and reliability of ad hoc scales, but they should strive to obtain and
report independent evidence of validity. For example, researchers might recruit
independent raters (e.g., colleagues or students) to read the items along with items
intended to reflect other variables, ask the raters to rate the clarity with which each
item reflects each of several variables, and evaluate the degree to which each item
was rated as clearly reflecting its intended variable. Such examinations produce
validity-related evidence that goes beyond the researcher’s opinions, and they
might convince readers that a scale is sufficiently valid for narrowly-focussed
application.
Modified Scales
In addition to creating ad hoc scales, researchers sometimes modify existing
scales for new purposes. For example, researchers might shorten an existing scale
or might revise a measure of one variable (e.g., health locus of control) to “fit” a
different variable (e.g., financial locus of control). Again, there can be good
reason for such modifications (e.g., existing scales are perceived as too lengthy,
or there is no existing measure of the variable in question).
However, there are caveats that arise with modified scales, paralleling those
associated with ad hoc scales and arising from the fact that modified scales might
not have the psychometric properties or quality of an original scale. Indeed, the
more substantially-modified a scale is, the less it can be assumed to have psycho-
metric quality similar to the original. Thus, one caveat is that well-validated,
original scales are preferable to modified scales. Because a modified scale’s psy-
chometric properties and quality might differ from those of the original scale, the
modified scale is—to some degree—an ad hoc scale. As such, its psychometric
properties and quality are unclear and suspect. Consequently, a second caveat is
that modified scales require psychometric evaluation and interpretation. Because
the psychometric properties of a modified scale likely differ from those of the
original, researchers should carefully examine the scale’s dimensionality, reliability,
and validity.
they preclude the use of internal consistency methods for estimating reliability (e.g.,
coefficient alpha). Because internal consistency methods are easier than most other
methods of estimating reliability, single-item scales are often used without atten-
tion to psychometric quality. This is a serious problem, preventing researchers,
reviewers, editors, and readers from knowing the quality of a measurement that,
being based upon single-item scale, is inherently suspect. Thus, researchers who
use single-item scales might examine test–retest studies in order to estimate the
reliability of the scales.
Importantly, brief or even single-item scales can have psychometric properties
sufficient for researchers facing strict constraints in measurement strategies. For
example, the Single-Item Self-Esteem Scale has good test–retest reliability esti-
mates, and it has strong convergent validity correlations with the widely-used
Rosenberg Self-Esteem Inventory (Robins et al., 2001). Similarly, the Ten-Item
Personality Inventory has good test–retest estimates of reliability for each of its
2-item scales, which have strong convergent validity correlations with longer
measures of their constructs (Gosling et al., 2003). The important point is that
brief scales are appropriate and useful when their psychometric properties are
adequate, as demonstrated by solid empirical estimates.
10
Difference Scores
One important topic addressed in this volume is the use of difference scores as
indicators of psychological phenomena. Difference scores—or “change scores” or
“discrepancy scores”—are obtained by measuring two component variables and
computing the difference between the two. For example, a participant’s intergroup
bias might be measured by having her rate the positivity of an ingroup and of an
outgroup, and then calculating the difference between the two ratings. The differ-
ence might be interpreted as the degree to which she has a more favorable attitude
toward the ingroup than toward the outgroup.
Difference scores are appealing, but their intuitive appeal masks complexities
that have been debated for decades. Although much debate highlights the sup-
posed unreliability of difference scores, there is also concern that difference
scores can lack discriminant validity, in terms of simply reflecting one of their
component variables. Both issues can compromise the psychological conclusions
based upon difference scores.
This volume discusses these complexities, including psychometrically-based
and statistically-based recommendations for handling them. First, and most gener-
ally, alternatives to difference scores should be considered seriously. That is, dif-
ficulties might be handled best by avoiding difference scores altogether, focussing
instead on their component variables. Second, if difference scores are used, they
should be used with attention to their components and to their psychometric quality.
There seems to be a tendency to ignore the fact that difference scores are variables
with psychometric properties that must be understood and considered when draw-
ing psychological conclusions. Without such understanding and consideration,
research based upon difference scores is ambiguous in terms of measurement
quality, statistical validity, and psychological meaning.
11
12
subsequent steps might produce a scale with poor validity and ambiguous
meaning.
In addition, researchers creating a new scale must articulate the context in
which it is likely to be used. The context includes at least two elements—the
likely target population and the likely administration context. Obviously,
the intended population will direct subsequent steps of item writing. For example,
the response formats, items, or instructions for a scale intended to be used with
adults would differ dramatically from those for one intended to be used with chil-
dren. As discussed earlier, researchers cannot assume that a scale developed for
and validated within one population is psychometrically comparable or similarly
meaningful in different populations. Similarly, the likely administration context(s)
must be considered carefully. For example, if the scale will be used primarily in
research contexts that are time-sensitive, then subsequent steps will likely focus
on brevity. Or, for example, if the scale will be administered via an online survey,
then researchers should consider implementing online strategies in Step 3 of the
construction process.1
13
14
variable. The psychological meaning of this is clear only when “a one-unit difference
in a predictor variable” is consistent across levels of that variable. This is true for
interval scales, but not for ordinal or nominal scales.
Regardless of ambiguities and disagreements, researchers generally treat
Likert-type scales (such as the hypothetical homophobia scale) as an interval level
of measurement. Particularly for aggregated scores obtained from multi-item
scales, researchers assume that scores are “reasonably” interval-level. For very
brief or single-item scales, this assumption is more tenuous. In such cases,
researchers should either consider alternative analytic strategies or acknowledge
the potential problem.
Summary
This chapter has reviewed principles and recommendation for scale construction,
evaluation, and use, and has summarized the scale construction process. The
remainder of this volume provides greater depth into the process, principles, and
practices, hopefully enhancing motivation and ability to pursue effective measure-
ment procedures.
Note
1 The possibility that online surveys differ meaningfully from traditional methods has
received some empirical attention. For example, Gosling et al. (2004) found that
web-based surveys produce results similar to those produced by traditional methods;
however, they note that “this question has yet to be resolved conclusively” (p. 102).
15