Professional Documents
Culture Documents
Questionnaire Design, Interviewing and Attitude Measurement
Questionnaire Design, Interviewing and Attitude Measurement
To
Betwyn New Edition
A.N. Oppenheim
CONTINUUM
London and New York
PREFACE
ACKNOWLEDGEMENTS
I INTRODUCTIONTO SURVEY DESIGN
2 ANALYTICSURVEY DESIGNS
3 DESCRIPTIVE SURVEY DESIGNS
4 PILOTWORK
Continuum
The Tower Building, 11 York Road, London SEl 7NX 5 THEEXPLORATORY INTERVIEW
370 Lexington Avenue, New York, NY 10017-6550 6 STANDARDIZED INTERVIEWS
0 A. N. Oppenheim, 1966, 1992
7 QUESTIONNAIRE PLANNING
8 QUESTION WORDING
First published as 'Questionnaire design and attitude measurement' by 9 SOMEBASIC MEASUREMENT THEORY
Basic Books, Inc., New York in 1966. This totally rewritten and recast
new edition first published in 1992 (reprinted 1993 and 1994) by 10DESIGNING ATTITUDE STATEMENTS
Reprinted 1996, 1997, 1998 (twice), 1999, 2000, 2001
11ATTITUDESCALING
1 2 PROJECTIVE TECHNIQUES IN ATTITUDE STUDY
British Library Cataloguing in Publication Data 13 VARIETIES AND APPLICATIONS
A CIP catalogue record for this book is available from the British Library
ISBN 1 85567 043 7 (HBK) 14 DATAPROCESSING
0 8264 5176 4 (PBK) 15 STATISTICAL ANALYSIS
Library of Congress Cataloging in Publication Data APPENDIXI: STATISTICS TEST
A CIP catalog record for this book is available from the Library of Congress APPENDIX11: NOMOGRAPHS FOR THE TESTING OF
All rights reserved. No part of this publication may be reproduced, STATISTICAL SIGNIFICANCE OF DIFFERENCES BETWEEN
stored in a retrieval system, or transmitted by any other means without PERCENTAGES
the prior permission of the copyright holder. Please direct all enquiries
to the publishers. INDEX
Typeset by DP Photosetting, Aylesbury, Bucks.
Printed and bound in Great Britain by Biddles Ltd
www.bidn'les.co.uk
From the preface to the first edition:
The world is full of well-meaning people who believe that anyone who can write
plain English and has a modicum of common sense can produce a good
questionnaire. This book is not for them.
It is intended to help all those who, for one reason or another, have to design
a questionnaire. It could serve as a textbook for research students and
undergraduates in social psychology and sociology and for practitioners of
G r k e t research. It could also help doctors, personnel officers, civil servants,
criminologists, social anthropologists, teachers, and many others whose curios-
ity or need for information may impel them into the quicksands of social
research.
Questionnaire design cannot be taught from books; every investigation
presents new and different problems. A textbook can only hope to prevent some
of the worst pitfalls and to give practical, do-it-yourself kind of information that
will point the way out of difficulties. It is not meant to be a work of scholarship,
nor is it exhaustive. The number of references has been kept to a minimum,
though some annotated readings have been provided with each chapter.
Throughout, clarity and basic ease of understanding have been my guidelines; in
each case, an attempt has been made to present the main lines of the argument
and, while not shirking weaknesses in underlying assumptions, to omit some of
the more esoteric controversies.
objectives have not changed. It still aims to reach those who have little or no
background in the social sciences, and those who lack the facilities which many
Western academics take for granted, as well as research students starting their
first project and the junior staff of social research agencies. Teachers of courses
in survey methods may find the book particularly useful because of the wide
variety of examples spread through the text. It is my hope that many will find
this not just a book to read but to keep.
In trying to make this text clearer, more accurate and better-written, I have
sought the help of colleagues and graduate students. I owe much to their incisive
critique and generosity of spirit and so does the reader. It is a pleasure to
acknowledge my particular indebtedness to Neville Postlethwaite, Roger
Thomas, Margaret Guy, Jennifer Marsh, Colin Taylor, Ronaldo Laranjeira, Julie
Parker and Dan Wright. The responsibility for the book's remaining faults must
be mine.
Bram Oppenheim I wish to thank a number of people for their kind permission to use material
London, January 1992 from their works, as follows:
A.L. Baldwin, J. Kalhorn, and F.H. Breese, 'The Appraisal of Parent Behaviour',
Psychological Monographs, XXIV (1949)~No. 299.
E.S. Bogardus, 'Measuring Social Distances', Journal of Applied Psychology, IX
(1925)No. 4.
William Caudill, T h e Psychiatric Hospital as a Small Society (Harvard University Press,
1958).
H.T. Himmelweit, A.N. Oppenheim, and P. Vince, Television and the Child (Oxford
University Press, 1958).
Michael Shepherd, A.A. Cooper, A.C. Brown, and G.W. Kalton, Psychiatric Illness
in General Pracfice (Oxford University Press, 1981).
The Open University. (Course DE.304)
1
INTRODUCTIONTO SURVEY DESIGN
Thus, by running backwards through the survey stages we can try to ensure from the lower income levels, or had larger families, were less religious and so
logic-tightness, so that nothing is left out and we shall have measured on. For a few points of comparison, data were probably available for the nation
everything that needs to be measured. as a whole (including the television owners), but most of the questions had to be
The formulation of any questionnaire to be used in a survey must be an asked afresh. In order to show how television buyers differed from non-buyers,
integral part of the research design stage. A questionnaire is not just a list of the second sample would have to be a representative sample of all the people
questions or a form to be filled in. It is essentially a measurement tool, an who had not bought a television set. That a sample of people living next door to
instrument for the collection of particular kinds of data. Like all such set owners would accurately represent the non-buying public was clearly too
instruments, the aims and specifications of a questionnaire stem directly from much to hope for, since in any case they had been chosen on the grounds of
the overall research design. Such objectives and specifications are not always similarity in environment and socioeconomic background to the set owners. As
obvious from a perusal of the questions. However, we cannot judge a the figures subsequently showed, the second sample, not being representative
questionnaire unless we know what job it was meant to do. This means that we of the non-television buyers, was useless for purposes of comparison. Conse-
have to think not merely about the wording of particular questions but, first and quently, the accurate figures obtained from the first sample could not be made
foremost, about the design of the investigation as a whole. to reveal any social change or pattern.
Thus it is essential for any researcher to draw up a statement (a) of general And what of the second purpose of the investigation; how was the new
aims for the proposed study and to develop these into (b) a set of specific aims; medium affecting set owners? Here there were many difficulties, since it is often
these should be further developed into (c) a set of operationalized aims; that is, hard to disentangle the effects of one variable, such as television, from the effects
specific issues or hypotheses, which should point directly to (d)a list of variables, of many others operating at the same time. A comparative study can try to do
scales or indicators which will be needed: for example, how shall we measure this by choosing a control sample as similar as possible to the television sample
parental education or strictness of upbringing? Dummy tables for the proposed in every relevant respect, so that we can see what the people in the television
analysis should also be drawn up at this point. sample would have been like if they had not bought television sets. There are still
weaknesses even then because of pre-existing differences, matching problems
and effects of guest-viewing, but the sample design failed to produce sufficient
An actual case similarity between the two groups. Consequently, there was no way of telling
whether or in what way changes had been produced in the television sample.
In the early fifties, when television-viewing was rapidly gaining ground in Great The second sample, that of the neighbours, turned out to be useless. It was
Britain, a survey was commissioned to supply answers to two broad questions: neither a representative sample of non-television buyers (as required for the
(I)what kinds of people were buying television sets? (2) How was the new first purpose) nor a closely comparable control sample (as required for the
medium affecting them? The large-scale survey that eventually took place used second purpose). It should have been realized from the beginning that, since the
two samples. The first was intended to be representative of television-owners inquiry had two objectives, two different comparison samples were required. This
and was drawn from the lists of television licence holders. This sample was may seem obvious in retrospect, but the professional survey organization that
bound to be incomplete since it failed to include set owners who had not applied carried out the research realized too late what had happened. Mistakes like this
for a licence, but by and large it could give reasonable estimates of the income, occur every day and can only be prevented by making the greatest possible effort
family size, social and educationalbackground, religion, consumer habits, and so to clarify our objectives before we start.
on of law-abiding television owners at that time.
The second sample was assembled by asking the interviewers to question an
adjacent non-television householder for each set owner in the first sample. It Surveys versus experiments
was reasoned that by interviewing the people next door a sample could be
obtained, for purposes of comparison, that would be similar to the television How do we decide which of the many types of research design to choose for our
owners in socioeconomic background, consumer habits and so forth. particular purpose? This is a difficult question to answer since every research
The faults in this design were only realized toward the end of the project presents its own particular design problems. Still, it is possible to discuss
investigation. Let us consider the logic of it step by step. The research had two some basic types of research design in general terms and to make some rough-
purposes. The first was to find out what kinds of people were buying television and-ready distinctions. In the next chapter we shall discuss cross-sectional
sets. This implied something more than a purely descriptive study of a designs, longitudinal designs, intervention designs, factorial designs and regres-
representative sample of set owners, since to state that such families had an sion designs. All of these (and several others) can be grouped together under the
average income of X pounds a week or that Y per cent of them had telephones heading 'analytic designs' (see Chapter 2) because, in one way or another, they
or washing machines would be almost meaningless unless it could be put into attempt to deal with associations or with cause-and-effect relationships in ways
some kind of context; in other words, it had to be compared to other figures. It analogous to lab experiments. They try to explain things, they seek to answer
might then become possible to conclude that television buyers more often came 'why' questions, or 'what determines X' questions and, in this way, they are to
12 Questionnaire Design, Interviewing and Attitude Measurement Introduction to sumey design 13
outcomes or the number of homes that will be required in ten years' time and so
be distinguished from the other broad category, the 'Descriptive' designs. (See on. Representativeness (or sometimes full enumeration) of a defined population
Chapter 3.) is a first requirement.
Beyond the world of survey research, the scientific endeavour is characterized
There are many questions that actuarial surveys cannot answer or can
by another type of analytic design, the controlled laboratory experiment. This , . -
answer only inadequately. Such questions usually start as 'why' questions and
method, which involves the systematic manipulation of variables, is not usually
then proceed to examine group differences from which relationshivs between
. ..
available to the survey researcher who cannot, say, make people want to adopt
variables can be interred. For instance, in comparing the results of election polls,
a pet to see what effect this will have on them or cause them to be burgled to see
we may find a rise, over a period of time, in the percentage of people choosing
how this will influence their attitudes towards the police. While it is sometimes a particular party. We may wonder why this is happening and wish to explore
possible to conduct simple experiments with people involving the manipulation the suggestion that, say, this party's policies with regard to old-age provisions
of several variables, it should not be assumed that the controlled laboratory makes it very attractive to older voters. We go back to our data and arrange the
experiment is the 'ideal' design. Experiments and surveys are often contrasted results according to age to see whether there are marked age-group differences
with each other. Those who favour surveys criticize experiments for being in the expected direction. We could go further and try to find out whether low-
unrepresentative, for dealing with artificial situations and for often failing to
income groups, or childless elderly couples, or couples with elderly
achieve the degree of precision and control that might justify them. Experimen-
dependents are more prone to vote for this party, but clearly there is a limit to
talists are critical of surveys because of their reduced ability to control or
what we can do in t h s way: we may not have obtained the necessary
manipulate important variables, for following events rather than making them information about income or elderly dependents; there may not be enough low-
happen and for their inability to prove causal relationships. Nevertheless, the
income voters above a certain age in our sample; we may not have asked any
survey researcher using an analytic design will have taken many a leaf out of the
questions bearing on pensions and so forth.
experimentalist's book. It would be more helpful to suggest that choosing the
We have now moved from questions about 'how many' to questions about
best design or the best method is a matter of appropriateness. No single approach is 'why', from enumeration and description to an analysis of causality, and so we
always or necessarily superior; it all depends on what we need to find out and on
shall require a different type of survey, a survey with an analytic design.
the type of question to which we seek an answer. Indeed many research
enquiries have employed surveys and experiments at different stages, using the
result of the one to inform and refine the other, and so producing conclusions
that are both precise and representative. Problems of causes and effects
Many current ideas about causality have their origin in what we think of as
'science' or the scientific method. All science is concerned with relationships or
Descriptive versus analytic designs co-variation between variables, but some of our ideas about relationships or
associations have to undergo subtle changes as we move from the physical and
As stated above, it is possible to make a broad distinction between two types of biological sciences to the social sciences. We may start with the 'monocausal' model,
survey: (I)the descriptive, enumerative, census-type of survey; and (2) the suggesting that a single cause can have a specific effect. Thus, if we put a kettle
analytic, relational type of survey. of water on a fire it will get hotter and eventually it will boil; similarly, the harder
The purpose of the descriptive survey is to count. When it cannot count it rains, the higher will be the level of a river and, eventually, it will overflow its
everyone, it counts a representative sample and then makes inferences about the banks. In the 'heroic' early years of microbiology the thinlung among researchers
population as a whole. There are several ways of drawing a sample, and the was often monocausal: one disease, caused by one strain of bacteria, to be cured
problems of sampling, sampling bias and sampling error are dealt with later (see by one vaccine orother anti-body. Associated with these monocausal or 'A
Chapter 3). The important point to recognize is that descriptive surveys chiefly causes B'models were several other ideas. For example, causality of this kind was
tell us how many (what proportion of) members of a population have a certain assumed to be linear, proportional and incremental: the stronger the cause, the
opinion or characteristic or how often certain events occur together (that is, are bigger the effect. In many cases this type of causality was also expected to be
associated with each other); they are not designed to 'explain' anything or to
reversible: if the cause were weakened or removed, the effect would decline and
show causal relationships between one variable and another.
disappear.
Descriptive surveys are well known and important. Any form of census falls
However, the monocausal model often does not tell the full storv. and
into this category, as do many public-opinion polls and commercial investiga- sometimes it can be seriously misleading. For example, the level of the ,river J ------
North Sea. Each of these causes will have a greater or lesser effect depending on are less well educated, on average; that people with less education tend to read
its strength and timing and so we have to learn to think in multi-causal terms. Not less; that in large families there is often neither the time to read nor the money
only that, but the effects of these causes may be singular or cumulative: when to buy magazines containing articles about upbringing and so on. Each of the
adverse conditions of wind and tide, moon and rainfall combine at the same time, associations is not just linked independently with maternal strictness; it is also
we may well get an exceptionally high water level and even flooding. Likewise, likely to be linked to the other associated factors and co-vary with them, as well
though the cholera bacillus is present in small numbers in most communities, it as with strictness. It would take quite a complex research design and statistical
does not usually become a dangerous epidemic unless certain other conditions analysis to disentangle these strands and to establish causal links from such a
obtain: poor hygiene, contaminated water supply, lowered resistance among network of associations.
certain members of the population. The nineteenth-century British statistician Sir Francis Galton once conducted
Let us also note that, in a multi-causal model, the several causal factors may an inquiry into the 'Objective Efficacy of Prayer'. He operationalized his inquiry
be independent or interrelated. Thus, the twice-daily tides and the phases and by trying to find out whether clergymen lived longer than members of other
diurnal orbits of the moon interact with each other, as well as with the water professions. (So far, by making a few assumptions, he appears to have followed
level in the Thames. In developing countries or under disaster conditions, poor a monocausal model: more praying causes longer life.) His data did indeed show
hygiene, lowered resistance and water contamination often go hand-in-hand to a small association: a difference of between one and two years in favour of the
create the conditions in which cholera flourishes. clergy. But at this point Galton seems to have adopted a multi-causal model of
Nor is it always necessary for these interactions among causes to happen longevity. He expressed the thought that such a small difference in average age
simultaneously. Their accumulation may take many years and form a succession attained might well be accounted for by easier life circumstances, concluding
of influences, for example the silting up of a lake due to industrial effluent, that 'the prayers of the clergy for protection against the perils and dangers of the
followed by the proliferation of algae and the death of fish and other forms of night, for protection during the day, and for recovery from sickness, appear to
marine life. Or the death of an individual from lung cancer may be due to a be futile'.
degree of genetic predisposition followed by a stressful lifestyle and prolonged Strictly speaking, though, even if GaltonS data had shown a very large
heavy smoking. This is sometimes called a causal pathway or, in people, a 'careerJ. difference in the average age attained by clergy compared to other professions,
Thus, a young person does not become a delinquent in a monocausal way, for this would not have constituted proof of a causal connection between prayer and
example, by coming from a 'broken' home. Other factors, successively or longevity. Some pre-existing, self-selective factor might well account for the
simultaneously, help to create a delinquent career: upbringing factors, educa- results; for example, it might be that in those days the clergy attracted
tional factors, friends and the 'delinquent sub-culture' and so on. exceptionally healthy recruits with a predisposition to longevity. O r the results
When trying to disentangle problems of causality we often find associations or might be explained by another, more current, association; clergymen might be
correlations, but of themselves these are no proof of causality. We might find, for sustained into old age by exceptionally supportive wives!
example, that people who have acquired a microwave oven also have poor We must also consider time, or rather temporal sequence, when seeking to
dietary habits, or that children with access to a computer at home are also better establish causality. Our understanding of causality includes the strong pre-
at mathematics in school. Does this mean that possession of a microwave or a sumption that causes must antecede their effects. Since it is implicit in this view
home computer caused the dietary habits or the maths performance? O n the that earlier causes are likely to be more important, this may lead us to discount
basis of such associations alone we cannot say. It might be true that A caused B; later causes. Thus, som&'women suffer from post-natal depression; but was the
or it might be that B caused A; that is, that people with poor eating habits are baby's birth the cause of the depression, or was the woman suffering from
more likely to buy a microwave and that children who are good at maths try to episodes of depression long before the pregnancy? We must not assume that
get their parents to buy a home computer. Or it could be that another just because B followed A, therefore B was caused by A. The baby's birth may, or
determinant, factor C, was the cause of both A and B. People with a busy may not, have been the significant event that caused the mother's depression.
lifestyle might have poor eating habits and also need a microwave; children from We merely picked up this association because the two events happened at about
better-educated families might both be good at maths and have access to a home the same time.
computer. We note, therefore, that associations may or may not denote To establish causality with any certainty is difficult; to establish associations
causality; that such causality might run from A to B, from B to A or from C to is often easier but leaves us to speculate about causality. It sometimes happens
both A and B. And often there is an interactive network of several causes. that association patterns can give us a strong hint about causality, and perhaps
Some mothers score hgher than others on a measure of strictness. We may suggest an effective intervention method, even though the cause remains
find that they are more often working class; that they have larger families, on unknown. John Snow, the London physician who worked in Soho in the
average; that they have completed fewer years of full-time education; and that nineteenth century, was the first to show the connection between cholera and
they are less likely to read women's-magazine articles about child upbringing. It a contaminated water supply. He meticulously mapped the addresses of those
would be quite possible to map out these associations and then speculate about who died from cholera during an epidemic at a time when micro-organisms were
causal interpretations. However, we shall also find that working-class women unknown and the beliefs about the causes of cholera were mainly superstitions.
16 Quesfionnaire Design, Interviewing and AfBfude Measurement Introduction fo survey design 17
His data showed that all those who had contracted the disease had taken their a prisoner escapes from gaol, the prison governor is sacked, although he
drinking water from the Broad Street pump. When he eventually persuaded the probably had no knowledge of the prisoner and no direct influence on the events
parish council to remove the handle from the pump, the epidemic stopped. All of the escape. This is done in the name of an attributional system called
Snow had, by way of evidence, was an association; discovery of the cholera 'accountability'. If a child commits a dangerous or criminal act, we demand
bacillus still lay in the future. However, disjoining the association had the desired restitution from the parents although they were not directly involved; we do
effect -though it has been argued that the result was only partly causal because this in the name of 'responsibility'. Such attributional systems have little or
the epidemic was on the wane anyway! nothing in common with valid cause-and-effect analysis, but simplistic mono-
Even today in medicine it is not unusual to find health measures being causal models and punitive attributions are easier to use than more appropriate
recommended for the cure or prevention of ills whose causes are as yet but more complex causal models, and they are culturally sanctioned and widely
unknown. The expected efficacy of these measures rests on the disruption of available. More appropriate multi-causal models, on the other hand, would
patterns of association, for example between coronary heart attacks and various demand resources, time and skills for their validation and further research to
dietary factors. Sometimes, too, the reverse happens: we may find an effective examine the effectiveness of appropriate intervention programmes. Thus the
remedy, even though we do not understand how it works, for example quinine social researcher who seeks to establish appropriate causal models and to
and malaria. validate them not only faces the daunting problems of research design but also
The laboratory experiment, if properly conducted, is the classical tool for encounters much societal resistance.
establishing causality. It does so partly by systematic manipulation of particular
causal factors (to show changes in causal factors being accompanied by changes
in effects) and partly by eliminating as many other factors as possible. Thus, pre- The future of causal attribution
existing causes may be eliminated by allocating subjects randomly to the various
experimental conditions; co-variation of causes may be explored systematically
Social research is trying to move from intuitive causal attributions (such as can
and interaction effects observed; and measures such as a double-blind design can
reduce the risk that human experimental subjects will influence the results by be found in scapegoating, collective representations, stereotypes, health-belief
systems, magic, folklore)to more objective kinds of causal attribution which are
their own expectations, or that the experimenters or their assistants will
unwittingly convey their expectations to their subjects. But critics of human capable of verification and generalization. We have come part of the way with
experimentation will often point to the difficulties experienced in creating true the aid of the research designs and strategies discussed above, but we still have
a long way to go. This is partly because many of our ideas about research design
experimental manipulation of psychological variables and to the fact that causes
are not always what they seem: in survey research we can develop complex and statistical analysis have come to us from other branches of knowledge, for
analytic designs to establish patterns of associations, but we can hardly ever example from the natural sciences, where other processes of causality predom-
prove causality because the experimental method is not usually open to us. inate and where the linear approach to measurement has often been of good
service. But societies and human beings are in important ways different from
However, we can sometimes participate in, or even help to plan, so-called natural
experiments or social interventions or intervention programmes which, if agricultural plots or biochemical reactions, and so we often feel that to apply
properly conducted, can permit us to draw causal inferences. some of these research designs to them is rather like putting our subjects on
Taking part in social intervention programmes confronts us with the fact that Procrustean beds.
most societies have strongly held world views and many systems of causal For example, we rightly seek to distinguish between 'mere correlates'
attribution. For example, many religions have built-in assumptions about the (associations)and 'actual causes', but the reason we cannot give associations the
causes of poverty, ill-health, crime and depression, and other assumptions about status of causes is because we do not know their place in a complex network of
what makes a good marriage or a happy childhood. Similarly, the authorities causality. The very word 'network' is a metaphor. It implies assumptions about
responsible for law and order hold strong assumptions about the causes of the ways in which multi-causality works but, in fact, we often cannof say what the
criminal acts, traffic accidents, football hooliganism or prostitution. Typically, correcf causal model would be. Take, for example, the long-term stability of belief
they will adopt a monocausal model, usually one embracing the personal systems and attitudes. Research shows that older people are often more
responsibility of an individual. Thus, the responsibility for a car crash will be religious than younger people and strongly believe in the hereafter. Does this
attributed to the driver with traces of a drug in his bloodstream. That the mean that they had a more religious upbringing because they were educated
long ago and have not changed since? Or have their religious ideas been subject
accident occurred on a notoriously dangerous bend, that the car crashed into a
poorly sited telephone pole, that the road was made slippery by rain or ice, that to many influences and changes throughout their lives? O r have they recently
the car's tyres were almost bald, that visibility was very poor at that time of 'found' religion because a strong belief system helped a close friend to come to
night, that the driver was blinded by the lights of an oncoming vehicle are all terms with impending death? Our models of attitude measurement and change
facts that are often ignored, requiring a multi-causal model which the legal seem too linear to reflect such processes. Again, we often find that liking a school
system would find difficult to handle. Societies often indulge in scapegoating:if subject and being good at it go together, but by what means or metaphors can
I 8 Questionnaire Design, Interviewing and Attitude Measurement Introduction to survey design 19
w e describe and measure how such a n association develops? The same might Kidder, Louise H., Judd, Charles M. and Smith, Eliot R., 1986, Research Methods in Social
Relations, Holt, Rinehart and Winston, New York.
apply to alcoholism and drink-related occupations; o r t o the stages of becoming Published by the SPSSI as a successor to the earlier text by Selltiz, Wrightsman, and
a criminal. Such processes are ill-served by our traditional causal models. Cook (see below), it is a comprehensive text at the undergraduate level.
We seem t o have inherited notions of causality which imply a linear, additive
progression, a kind of geological or 'layered' causality in which bounded one-way Moser, C.A. and Kalton, G., 1972, second edition, Sumey Methods in Social Investigation,
influences are brought to bear o n people at successive times causing them t o Heinemann, London.
change. O u r designs and our statistical techniques predicate us t o think in this A well established, statisticallyorientated general text in survey methods, especially
strong on sampling methods but also good on interviewing, questionnaire design
way because w e know how t o apply them; this kind of causality w e can more or and scaling methods.
less handle. But in social research w e often come across what might be called
'spiral reinforcement' processes. A small success at school, or a word of praise Hoinville, Gerald and Jowell, Roger, 1978, Sumey Research Practice, Gower, Aldershot,
from a teacher, may make a child just that little more interested in a particular Hants.
school subject, the child tries harder and develops its interest in that subject, Straightforward and practical general text on survey methods. Especially good on
organizing fieldwork and postal surveys.
which in t u r n may lead to more praise and more success and so on. T h e process
is incremental but not additive, influential but also interactive. The same may Babbie, E., 1989, second edition, Sumey Research Methods, Chapman & Hall, London.
apply to the drinker w h o becomes a barman. A pre-existing liking for drink may Kerlinger, F.N., 1964, Foundafions of Behavioural Research, Holt, New York.
lead by mutually reinforcing steps - a self-selective job choice, easy availability A comprehensive research methods text, especially strong on measurement and
of drink, increased liking, membership of the drinking scene - t o the statistical problems.
development of problem drinking and so, ultimately, to dependence and
Hippler, H.J.,Schwarz, N. and Sudman, S. (eds),1987,Social Information Processing and Sumey
alcoholism. As yet w e have no models that can display, nor research designs that Methodology, Springer Verlag, New York.
can reveal and predict, the workings of such causal processes.
In choosing our research strategy, our research design and statistical analysis, Maclean, M. and Genn, H., 1979, Methodological Issues in Social Sumeys, Macmillan, New
w e should therefore remain aware that these contain implicit assumptions about York.
causal links and causal processes in people. Indeed, such assumptions may long Schuman, H. and Kalton, G., 1985, 'Survey methods' in Lindzey, G. and Aronson, E.
ago have insinuated themselves into our own research interests and into our (eds), Handbook of Social Psychology, Vol. I, third edition Random House, New York.
thought processes about causal attributions. While w e can make good use of An extensive textbook chapter touching on all the main points of survey research.
existing research methods in the service of replicability, data disaggregation and Altreck, Pamela L. and Settle, Robert B., The Sumq Research Handbook, Richard Irwin Inc.,
representativeness, w e must not forget that human lives and human causality Homewood Ill.
are not composed of layers of regression coefficients. We need a fundamental re- An easy to read undergraduate text.
think of the nature of the social influencing processes themselves. This may lead
t o the development of more appropriate analytic designs and t o better models of Singer, Eleanor and Presser, Stanley, 1989,Sumey Research Methods, University of Chicago
Press, Chicago.
human causality. A reader dealing with sampling, non-response problems, validity, interviewer bias,
telephone interviewing and related topics.
Przeworski, Adam and Teune, Henry, The Logic of Comparative Social Enquiry, Wiley
Selected readings (Interscience),Chichester, Sussex.
Deals with the logic of comparative studies and the problems of equivalence of
measures. For advanced students.
Sage University Papers Series: Quantitative Applications in the Social Sciences.
An expanding series of small paperback volumes dealing with every aspect of social
Rossi, P.H., Wright, J.D. and Anderson, A.B. (eds), 1983, Handbook of Sumey Research, research methods.
Academic Press, New York.
A good, comprehensive collection. The Open University, Course DEH 313, 1993, Principles of Social and Educational Research,
The Open University, Milton Keynes.
Kidder, Louise H., Judd, Charles M. and Smith, Eliot R., 1986, Research Methods in Social An excellent set of self-study course books, available separately and supported by
Relations, Holt, Rinehart and Winston, New York. exercises and by collected readings.
Published by the SPSSI as a successor to the earlier text by Selltiz, Wrightsman, and
Cook (see below), it is a comprehensive text at the undergraduate level. Fink, Arelene and Kosecoff, Jacqueline, 1985, How to Conduct Sumeys, Sage, London.
A text for beginners.
20 Questionnaire Design, Interviewing and Attitude Measurement
Selltiz, C., Wrightsman, L.S. and Cook, S.W., 1976, third edition, Research Methods in Social
Relations, Holt, Rinehart and Winston, New York.
Still generally useful.
Hyman, Herbert, 1955, Sumey Design and Analysis, Free Press, Glencoe, Ill.
A major older research text full of useful applications. Helpful on descriptive versus
analytic designs.
Bvnner, John and Stribley, Keith M. (eds), 1978, Social Research: Principles and Proceduw,
~ b n ~ m and
a n The Open University Press.
A useful collection of original papers on research design and methods of data
collection.
1. Experimental variables. These are the 'causes' or predictors, the effects of which
are being studied. They are sometimes referred to as 'independent' or
'explanatory1variables. The analytic type of survey, like the lab experiment,
is set up to vary these factors systematically so that their effects can be
observed. Often several such variables working both in isolation and in
various combinations are of interest.
2. Dependent variables. These are the results, the effects-variables, the gains or
losses produced by the impact of the experimental variables, the predicted
outcomes. These variables have to be measured particularly carefully and
group differences tested for statistical significance.
3. Controlled variables. As a source of variation these should be eliminated in order
to fulfil the condition of 'other things being equal' when the effects or
correlates of the experimental variables are stated. Variables can be
controlled by exclusion (for example, by having only males in a sample, gender
is excluded as a source of variation); by holding them constant (for instance, by
interviewing all respondents on the same day, thus eliminating day-of-the-
week effects);or by randomization (for instance, in the case of a multiple-choice
question, by systematically randomizing the order in which the alternatives
are presented to the respondents, thus eliminating ordinal and serial effects
as a source of variation).
4. Uncontrolled variables. These are 'free-floating' variables and can theoretically
22 Questionnaire Design, Inferuiewing and Attitude Measurement Analytic suroey designs 23
be of two kinds: (a) confounded variables and (b) error. The confounded variables, known to be randomly distributed, can bias the results to an unknown degree.
sometimes called 'correlated biases', have hidden influences of unknown size The 'ideal' design would contain no uncontrolled variables.
on the results. For example, in medical research the results of treatment may For our design to be effective, each of the four types of variable must be
be affected by a hitherto unknown allergy or by unsuspected side effects; in measured as carefully as possible. This is of particular importance with regard to
some psychological research, genetic influences may play a much bigger part our dependent variable(s). It often happens that, in designing a study, people
than anticipated; in advertising, public reaction to a new product or campaign become so involved with the problems of the experimental variables (the
may be unexpectedly volatile. Essentially this means that knowledge and 'causes') or the controlled variables, that they give insufficient attention to the
understanding of the phenomena under investigation are still incomplete in dependent variable - or even forget to measure it altogether! Thus, in the
important ways; there are variables, other than the experimental and earlier example about children's bedtimes, they may simply forget to ask any
controlled ones but confounded with them, that can affect the results and questions about bedtimes - without which all the other questions are useless.
hence can produce serious misinterpretations. O n the other hand, such It will also soon become clear that 'bedtime' is not a precise concept: does it refer
uncontrolled variables can lead to the development of new hypotheses so to the 'official' bedtime (if any) laid down by the parents, to the time the child
that, eventually, their impact may be controlled. actually gets into bed, to the time the child falls asleep? Do we ask the child, do
Inevitably, any research design also suffers from error. Such error we ask the parents or perhaps an older sibling, if any?
variables are (or are assumed to be) randomly distributed or, at any rate, In deciding how to measure such a variable, we first have to define it and then
distributed in such a way as not to affect the results. define it further in operational terms. This process of defining should take us
right back to the initial aim or purpose of our research. If we are, say, conducting
In practice it is not usually possible to distinguish between confounded some kind of medical enquiry about the need for rest and sleep in children, then
variables (that is, hidden additional causes) and 'pure error'. In analytic surveys, 'bedtime' might well be defined as 'falling-asleep time' (to be followed by
as in experiments, the influence of uncontrolled variables is made as small as 'waking-up time' and questions about interrupted sleep).But if we are interested
possible. If the presence of confounded variables is suspected, a good deal of in, say, social-class differences in parental standards, then 'bedtime' might be
statistical analysis may be required to uncover their identity, and ultimately their more appropriately defined as the 'official' bedtime (if there is one). Thus, the
i m ~ a cmust
-- t be studied svstematically in a new enquiry. definition of a variable will not depend on a dictionary but on the internal logic
i n example may help to clarify these terms. Let us assume that we are trying of the inquiry.
- - understand
to -- - - - - - some of the determinants of children's bedtimes. We have decided- Further pilot work will then show how a variable, as defined for our purposes,
to study the effects of age, which will be our experimental variable, and the can best be measured in the field. The pilot work may even show that we need
survey will obtain information from a sample of children for every day of the still further refinement in our definition. We should never shirk any effort to
week. Our deoendent variable will be the children's bedtime, in all its variations.
1.
make our dependent variable more precise and more robust. If we had simply
There are many variables that we will have to control in order to observe the asked a sample of parents about their children's usual bedtimes, then it would be
effects of age. For instance, children who have recently been ill may have to go very difficult to draw conclusions about differences in age, gender, school term
to bed especially early; we will probably control this variable by excluding such versus vacation, week-ends etc. because parents may have interpreted our
children from the sample. Children go to bed later in the summer and during question in several different ways -with such an ill-defined dependent variable
school holidays; we may control these variables by holding them constant, by the rest of the study becomes almost meaningless.
collecting all our data during one short period in the school term. Children go to Often there are -difficulties in trying to &fine and measure a dependent
bed later if they have older brothers or sisters; we can try to take care of this by variable. Imagine, for example, a study of the determinants of recovery from
making sure that children with older siblings are randomly distributed through alcoholism; the term 'alcoholism' is difficult enough in itself, but to define
our sample. 'recovery' from it is more difficult still. O r suppose you have been asked to
There remain a considerable number of uncontrolled variables, some of which measure the effects of a health education campaign, say with regard to the
are very likely to be confounded. For instance, socioeconomic background is prevention of coronary heart disease. What exactly would be your dependent
likely to be an important influence on bedtimes; if we realize this in advance, we variable? A measure of knowledge? A measure of 'acceptance' or 'awareness' of
may be able to control it, but otherwise this factor can easily introduce a bias in the campaign's message? A set of cause-of-death statistics? Data from all the
our conclusions. It may, for instance, happen that the older children in our local hospitals?
sample also come from more well-to-do homes, where bedtimes tend to be Or again, suppose you have been called in by a municipal transport authority
earlier; in that case, we may wrongly conclude that increasing age is not a good to find out whether the painting of bus lanes on certain busy roads (into which
predictor of later bedtimes because, unknown to us, the socioeconomic factor is cars are not allowed between certain hours) is doing any good. What would be
counteracting the age factor. Again, the child's gender, membership in a youth your dependent variable(s)? Average bus speeds before and after? Accident
organization, or a keen liking for television may be important determinants of rates? Delays to motorists? Police expenditure? How, furthermore, would you
bedtimes in parts of our sample. Such uncontrolled variables, unless they are try to measure each of these variables free of bias? And how would you make
24 Questionnaire Design, Interviewing and Attitude Measurement Analytic survey designs 25
allowance for different times of day, different days of the week, different classes the proportion of working-class children was much higher among the urban
of road users, inbound versus outbound traffic flows and the novelty impact of part of our sample. Perhaps, if there had been as many middle-class children as
such an intervention? working-class children in both the urban and the rural parts of our sample, the
urbanlrural differences would have been negligible.
Another point we note is that it becomes necessary to plan in advance the
Cross-sectional designs comparisons we wish to study and to make them part of our research design; we
cannot usually apply this kind of analysis to a sample which has been collected
Earlier we compared the bedtimes of different children at different ages, on the for some other purpose.
assumution that these comparisons could tell us what would happen - -children
- - to
over aAperiodof time. We did not actually take the same group of children and
follow them over a number of years. We hoped that the groups of children were Factorial designs
comparable and that we could thus in fact observe the relationship between age
and bedtimes. This type of design is known as cross-sectional to distinguish it The problem of interrelated independent variables has led to the development of
from longitudinal or 'before-and-after' designs. factorial designs, whose particular function is to disentangle complex sets of
Suppose that in our example we had obtained a probability sample of children, interrelationships. They are analogous to the 'agricultural plots' type of design
say, between the ages of five and fifteen. What often happens is that such a which led to the development of analysis-of-variance techniques. Analysis of
sample is collected for a descriptive or actuarial purpose and that a kind of cross- variance is, however, chiefly applicable to quantitative variables, whereas
sectional design is imposed on it afterward. First, we obtain the over-all average surveys deal mostly with categorical variables (see Chapter 9 ) .
bedtime. Next, we compare the different age groups and find that bedtimes get This type of analytic survey design is one of the ways in which we can
later with increasing age. Then, we decide to compare the sexes, and we find that approximate laboratory conditions. However, in laboratory experimentation we
boys, on average, go to bed a little later than girls. We go on in this way, create or introduce our experimental variables and, while controlling for most
comparing bedtimes against a number of variables in turn such as social class, other variables, observe the concomitant changes in the dependent variable. In
size of family, urbanlrural differences and so on. Suppose that each time we find survey research we are not normally in a position to impose experimental factors
a difference in bedtimes. We now want to go further; we want to know, for or to manipulate the lives of our respondents, for instance by allocating them at
instance, whether the age differences still remain if we hold social class constant. random to an experimental or a control group. Instead, we select respondents
This means that for each of our social-class groupings, we have to compare the who already have the characteristics required by our design and compare them
bedtimes of children in their different age groups separately. We could go still in their groupings. This makes it impossible to give causal interpretation to any
further and try to study the effects of three variables in combination, say, social pattern of associations we may find. Instead of dealing with just one
class, urbanlrural and size-of-family differences. Howeger, to do this we need to experimental variable and controlling all others, the factorial design enables us
have a sufficient number of cases in each of our combinations or 'cells' (unless to study several experimental variables in combination. Not only does this provide
our predictions refer only to part of the sample). Some of these cells will be us with more information but also with greater confidence in predicting the
readily filled, for instance those for children from large families in urban districts results under various circumstances.
with working-class backgrounds. Other cells wdl be difficult to fill or may At the outset, we have to decide which are going to be our experimental
remain empty, such as those for children from small working-class families in variables - we will want to vary these systematically and in combinations with
rural areas. This is because a sample selected to be representative of the child each other. We must aim to control all other variables by exclusion, holding
population as a whole will not contain sufficient numbers of these rarer constant or randomizing. The choice of experimental variables predetermines
combinations to make comparisons possible. what we can hope to get out of our study; if we decide, later on, that some other
We have moved to a different kind of research problem. Now we no longer variable should have been chosen, it may not be possible to show its effects
ask 'How many?' but 'What goes with what, and why?' We have moved from a except by selecting a new sample. This means that we must have an adequate
descriptive to an analytic type of survey that tries to answer questions about general idea beforehand of the lay of the land in our particular area either
relationships and determinants, and therefore we shall need a different type of through previous studies or from the pilot work.
sample, a sample that is primarily geared to making comparisons. The choice of our experimental variables will be limited by practical
Upon re-examining the preceding example, we note that our difficulties arose considerations. We might start off rather ambitiously by choosing, say, sex, age,
partly from running out of cases to fill the cells representing rare combinations class, urbanlrural and size of family in our bedtime study. Let us assume that we
and partly from the fact that some of our experimental variables were related to have ten age divisions, seven socioeconomic grades and five family-size
one another as well as to the dependent variable (bedtimes). Thus, in finding groupings, while sex and urbanlrural are dichotomies. This would give us a
differences between children with urban or rural backgrounds, we wondered design containing 10x 7 x 5 x 2 x 2 = 1,400 cells, wluch, even at no more than
whether this difference would disappear
- - if we held social class constant because ten cases per cell, would require 14,000 children. If this is unacceptable, we can
26 Questionnaire Design, Interviewing and A f t i f u hMeasuremenf Analytic sumey designs 27
to parental support with homework, to size of school class and so on; it is easy
either cut down the number of experimental variables, or make them cruder (for to think of two dozen or more possible determinants -far more than a factorial
example, by reducing class from a seven-point scale to a dichotomy),or probably design could cope with. Each of these variables contributes to the outcome (the
both. In this way we can reduce the number of cells and, thus, the number of dependent variable 'achievement in arithmetic', as tested) to a different degree
cases required, to more manageable proportions. Even so, a design requiring and in a different way. Also, we shall find that each determinant (experimental
three dozen cells and forty cases per cell would not be at all unusual. Equal (or or independent variable) may be related to others; there may be more effective
proportionate) numbers of cases are required in each cell in order to disentangle teaching in smaller classes or more parental support for more intelligent
the experimental variables from each other. Respondents in each cell should be children. This makes it difficult to find out what influence each independent
a probability sample (Chapter 3) of all those eligible to be in that cell. variable, of itself, has on the outcome or what would be the effect of changing
A disadvantage of the factorial design is the need for careful selection of cases. a particular independent variable.
Sometimes this will mean that a special enumeration sfage, using only a short There are advanced statistical procedures, such as the various kinds of
questionnaire, will have to precede the main inquiry (the much larger multivariate analysis (for example multiple regression analysis, path analysis),
enumeration sample being used to supply the respondents to fit each cell of the which are capable of disentangling t h s kind of associational network. Ideally,
factorial design). At other times, the number of respondents in some cells will they can do three things for us. First, they can give us a firm idea of the total
unavoidably remain below requirements, and in that case, some method of variance accounted for by our independent variables (that is, how much of the
statistical extrapolation or 'randomized replication' will have to be used. .
variation in our dependent variable we can 'explain' with the aid of the
The factorial design makes no pretence of being representative. Indeed, in its independent variables we have chosen and how much variation remains
final form it is often markedly unrepresentative, owing to the need to unexplained). Second, they can tell us which are the most important determi-
incorporate equal or proportionate numbers of respondents in each cell, so that nants and which are less important or insignificant. Third, they can tell us how
very rare combinations of determinants may be encountered as often as the powerful each determinant is after its links with other variables have been
more popular or typical ones. (However, it is sometimes possible to extrapolate discounted (for example, do brighter children still do better at arithmetic, even
and adjust the figures afterwards, using 'correction factors', to make this kind of when we 'allowJ for parental help with homework?). In other words, they can
sample yield estimates also.) tell us how powerful each determinant is, 'other things being equal', or 'holding
The factorial design permits us to vary our experimental variables systemat- everything else constant'. Sometimes, too, these statistical techniques can show
ically and in combinations of two or more, though such patterns of multiple what additional influence or strength each successive independent variable has
association cannot be interpreted as causal. We also need to introduce measures added during the emergence of a causal network or what its unique explanatory
to control the remaining variables and to eliminate as many uncontrolled power is.
variables as possible. Some of this can be done, as before, by exclusion, holding These multivariate techniques are not always applicable. They rest on
constant and randomization, but inevitably some confounded variables may statistical assumptions which may be hard to fulfil and generally they require
remain. In the later stages of the analysis, various hypotheses concerning these interval-type variables; that is, quantitative integers such as scale scores.
confounded variables will be explored by means of special cross-tabulations, Nominal or categorical data such as eye colour or marital status are more
matching of subsamples (see below), and by multivariate analysis. While a difficult to handle (see Chapter 9).
confounded variable may turn out to have an unexpected effect on the results, Moreover, even these more powerful techniques do not, strictly speaking,
upsetting the regular pattern of the design as planned, it can also be stimulating allow us to make causal references. Rather, they help us to disaggregate the
to further research and the discovery of new relationships. variance (the fluctuations or differences) in the dependent variable according to
the relafive importance of the independent variables which we have entered into
the design and to do this by statistical rather than by experimental means.
Designs using regression and other multivariate analyses Multivariat analysis is not for the beginner. You will need access to a
reasonably powerful computer, to the relevant computer software and to sound
As we have seen, in social research we rarely deal with monocausal phenomena, statistical advice. These techniques are best applied in well-researched domains
that is with a single cause having a specific effect. Almost invariably we have to such as education, in which it is possible to generate precise hypotheses and
deal with multi-causal models, so that any effect is the outcome not of one cause potential causal models about interrelationships between variables as well as
but of a complex network of determinants. Quite possibly many of these will not adequate measures by which to test them.
only be related to the dependent variable but also to each other: they will form One technique of multivariate analysis is called rnuNiple regression. What will
a network of inferrelated deferrninanfs. this technique do for you? First, having listed all your hypotheses and
Suppose that we are interested in some aspect of school achievement: why do constructed your measures of the dependent variable(s)and of, perhaps, several
some children do better than others at, say, arithmetic? We shall soon find that dozens of (potentially interrelated) independent variables, you will need to
this is not merely because some children are more intelligent than others, but obtain your data. You will not need a preliminary enumeration survey such as
because their performance is also related to several aspects of teaching at school,
28 Questionnaire Design, Interviewing and Attitude Measurement Analytic survey designs 29
on the other hand, would be appropriate when you are trying to discover the
might precede a factorial design study; but you will need substantial numbers (or main underlying determinants of your data or to test a set of hypotheses about
perhaps even a national sample of the relevant groups) to obtain adequate them.
variability on your measures and to enable you to show that any group In choosing the appropriate statistical technique much will depend on
differences you may find have not arisen by chance (that is, are statistically previous research, on how well understood the particular domain of enquiry is.
significant), and are large enough to be socially relevant and useful. After In some fields we have a number of reliable quantitative measures, some precise
processing and analysing the data the computer program will produce a table theories and some useful causal models and so we may be able to apply
which lists all your independent variables one by one and will give a figure for multivariate techniques of analysis with advantage. In other fields we would
each such variable to indicate how much it contributes to the 'explanation' of hardly know where to start, and the use of such analytic techniques would be
your dependent variable. A total will also be given, showing how much of the premature. What would we do, for example, if we were asked to find out why
variance is explained by all your independent variables put together. If your some people become amateur clarinettists or start to invest in the stock market,
measures and hypotheses have been strong, then together they might account while others do not?
for, say, 80 or 85 per cent of the total variance. In the social sciences we are more
likely to have poor predictors, so a total of 40 or 50 per cent is more probable -
leaving about half the variance unaccounted for. Left to its own devices, the
program will produce this first table showing your independent variables in Bef ore-and-after designs
order of the size of their explanatory contribution. Thus, this first pass through We have noted repeatedly that cross-sectional, factorial and multivariate types
the data will give you two things: (a) the total proportion of the variance of design cannot determine true cause-and-effect relationships; they can only
accounted for; and (b) a list of coefficients, starting with a handful of significant give us information about associations or correlates -though some of these are
ones and ending with a larger list of variables which have virtually no 'power', very suggestive! Before-and-after designs (sometimes referred to as pre-testlpost-
that is which at first sight
- contribute little or nothing to an 'explanation' of the test designs) have been developed in an effort to overcome this disadvantage.
dependent variable.
. .
You mav,- however, wish to go further. An advantage of multiple regression
.I
The form of the simplest before-and-after design is indicated by its name: a set
of measurements (base-line measures) is taken of a group of respondents, who
is that you, the researcher, candetermine the order or sequence in which your are then subjected to an experimental variable and afterwards measured again
variables will enter the analysis. You may have testable hypotheses about the - once more or perhaps several times. The difference between post-test and
relative importance of each independent variable and feed them into the analysis pre-test results or observations is said to be the 'effect' of the experimental
in this order. Or you may wish to test a general model. For example, in variable, though this is misleading.
educational research you might wish to determine the order of entry of your At first sight it may appear that this design will present far fewer problems in
independent variables according to the developmental stages in a child's life: respect of controlled and uncontrolled variables and error because the respond-
first, the pupil variables (such as IQ and quality of the home); then, the school- ents in the experimental group 'act as their own controls'. They provide the
related variables (such as average expenditure per pupil); after that perhaps the 'other things being equal' conditions that enable us to isolate the effects of the
teacher-related variables (for example, proportion of teachers with relevant experimental variable on the dependent variable so that we can draw valid causal
specialist training), and finally the classroom-related variables (such as size of conclusions about the process.
class, availability of equipment) - all depending on how the research design has But can we? We cannot legitimately attribute all the before-and-after
been set up and what questions it is trying to answer. differences to the effects of the experimental variable we are investigating until
While there are advantages in this flexibility, a word of caution is needed. This we are sure that, without it, such changes would not have occurred or would
is because many of your variables are likely to be interrelated, and so the impact have been smaller or different. Depending on the time interval between pre-test
of any given variable may be shared by that of others. This makes the results and post-test (a few hours, a day or two or perhaps many months), it is always
more difficult to interpret. It all depends on the type of associational network possible that some changes in the expected direction may take place even
you are studying and on the degree of independence of your variables. without the impact of the experimental variable. For example, some of the
This discussion of regression analysis has taken us rather far afield, yet it is people whom we have subjected to an advertising campaign to buy a certain
only one of a number of multivariate analysis methods. We might, instead, have product or brand might have done so anyway; some patients who have been
chosen to use analysis of covariance, factor analysis or some form of given an experimental treatment might have recovered even without this
discriminant function analysis. These all require more advanced statistical intervention. This may be due either to concurrent influences (for example a
knowledge (as well as the availability of appropriate computer software), and price reduction) or to pre-existing ones (for example a strong immune system).
your choice will depend on the kind of problem you are investigating. For Moreover, if people are aware that they are participating in a survey or an
example, cluster analysis or discriminant function analysis are probably most experiment, this in itself will often produce certain changes - they become
appropriate when you are seeking to develop a typology, a way of classifying more aware or alert, they develop expectations about the outcome, and with all
your subjects into a few distinctive sub-groups. Factor analysis (see Chapter 9),
Questionnaire Design, Intemiewing and Attitude Measurement Analytic survey designs 31
30
strictly controlled during the allocation of people to groups: preferably subjects
the attention they are receiving they may try to 'respond' extra-well. In medical should know as little as possible about the topic in question, they should not
research this is known as the 'placebo effect', where patients may respond know until later which of them was in the experimental group, and the
positively even when given an inert substance. allocation to each group should be done with the aid of a table of random
It is therefore necessary to have, in addition to the experimental group, a numbers (see below).In real life this is not easy since people often object strongly
matched control group. Indeed, sometimes we need more than one control group. to random allocation procedures. For example, if we need to evaluate the
To return to our earlier example, if we are investigating the effects of a new amount of social work and probation service care needed for ex-prisoners, we
medical treatment for a particular illness, we cannot simply count the number of might well decide to allocate subjects at random to a high-care and a normal-care
recoveries and attribute all of them to the new treatment; we need to know how (that is control) condition, but some ex-prisoners might have strong objections
many of our treatment group of, say, fifty-three patients would have recovered to being allocated to the control group if they become aware that they are part
anyway, and it is the function of the control group to tell us that. By subtracting of an experiment. There may also be ethical objections.
the number of recoveries in the control group from the number of recoveries in Special attention will need to be paid to the choice and measurement of the
the experimental group, we obtain the net effect of the treatment. dependent variable. Even in medical research it is not always clear what
We have now arrived at the basic outline of the classical 'effects' design. To constitutes a 'cure' or for how long a patient needs to be followed-up before we
start with, we need two matched groups as similar as possible. One will be can be sure that there will be no relapse. If we decide to study the effects of
designated the experimental group, the other the control group. At the 'stress', what measures should we use: psychosomatic illness episodes, subjective
beginning of the study, both groups are given base-line measures (the 'before' stress assessments, psychiatric breakdown symptoms, or what? If we decide to
condition or pre-test). Next, the 'treatment' or 'interventionr is applied only to run a campaign to combat the spread of sexually transmitted diseases by trying
the experimental group. The control group is given nothing, or only a placebo, to persuade people with certain symptoms to come forward for treatment, how
and is allowed to get on normally with its life. At the end of a suitable time can we measure the success or otherwise of our campaign? An obvious measure
period, both groups wdl be given the same measures again (the post-test might be the number of new cases that come to the clinics, but this might be a
measures). Next, we calculate the before-and-after differences for each group. short-lived effect. A subtler measure of 'success' might be the proportion of
Let us say that we find eighteen recoveries in the experimental group, and six cases who are willing to give names and addresses of sexual partners - which
recoveries in the control group. By subtracting the latter number from the might suggest that the campaign's 'message' has got through and might indicate
former we arrive at the net effects of the intervention; that is, twelve recoveries a longer-lasting effect (though this would need checking). If necessary our
which would not otherwise have happened. design will have to incorporate several different measures of effect and might
But the results of such a design are not always what they seem, and great care also need to incorporate repeated effects measures over a long period of time,
is needed in interpreting the outcome. One problem is that of the placebo effect. that is a long-term follow-up L i g n . The longer the time period, the harder it
In early trials of an experimental medication the psychological effect of being becomes to assign changes to causes because of the many intervening variables
given a new drug is often very positive, making it difficult to isolate the effect of that might affect the dependent variable - and, of course, there may be
the medication itself. Therefore, to make them feel part of the experiment, significant case losses during the follow-up period. If this happens, then the loss
members of the control group are given some inert substance (a placebo) which of numbers is less important than the ~ossibilityof bias, for example that we lose
they believe to be the new medicine. Even this may not be sufficient. It has touch with our treatment 'successes' while the 'relapses' remain with us.
sometimes been found that the nurses or doctors unwittingly make patients Even when we have a straightforward and readily measured dependent
aware that they are getting either the 'real' medicine or a placebo and so, variable, care needs to be taken in assessing effects. TOtake a hypothetical case,
nowadays, we adopt a'double blind' design, in which neither the patients nor the suppose we are conducting a political campaign to persuade an electorate to vote
medical staff know who is being given the new medication and who is getting for party X in a two-party system, and suppose further that voting intent or
the placebo, thus enabling us to isolate the drug's effects from those of other party preference both before and after our campaign is 50-50. Does this mean
variables. that our campaign has had no effect? Not necessarily. It might just be that every
There are other problems with before-and-after designs. How, for example, voter in our sample has changed to the other party, leaving the distribution of
can we be sure to get rid of pre-existing determinants? We might, say, wish to party preferences still at 50-SO! This leads us to the important observation that
find out the impact of a particular video presentation and so we get two similar comparisons between overall distributions before and after an intervention
groups and show the experimental group the video in question, while the leave us in the dark as to the amount of change that has actually taken place. In
control group is shown some other tape. But how exactly did we obtain the two the above example, if party X preferences had gone up, say from 50 per cent to
groups? Could there have been a 'volunteer bias', where subjects allocate 65 per cent, does that mean that 15 per cent of our respondents changed their
themselves to the experimental or the control group, perhaps because they are minds? Such a finding might be a serious under-estimate of the amount of
more, or less, interested in the topic? change that has actually taken place. For example Party Y might have lost 20 per
In such a case, this pre-existing interest factor might bias the results, probably cent of its initial supporters but gained S per cent of new supporters who had
by exaggerating the effects of the video tape. Volunteer error must therefore be
32 Questionnaire Design, Interviewing and Attitude Measurement Analytic survey designs 33
that both groups contain the same proportion of women, that their average age
come across from Party X - so the actual movement or change in party is about the same, that the proportion with post-secondary education is the same
preferences might be greater than the overall figures suggest. To overcome this and so on (group matching). An improvement on this is individual matching: for each
problem we should not rely on before versus after overall distributions; we
member of the experimental group we select another, very similar, individual to
should calculate a change score for each individual. Although change measures are
act as control. Since we obviously cannot match pairs of individuals on
often unreliable and difficult to manage statistically, they will give us a better
everything, we should match them on those variables which are relevant; that
indication of the amount of change that has taken place. is, are likely to be related to the dependent variable. In a well-studied field we
Sometimes we may become involved in assessing change processes beyond
should know on what attributes to match (though there may still be practical
our control. For example, administrative or political reforms are often intro- difficulties), but in other domains we have only common sense and perhaps
duced with scant regard for the requirements of accurate research design, and some pilot work to guide us. The situation becomes even more difficult when,
their assessment and evaluation is fraught with both experimental and political
as often happens in surveys, we have numerous dependent variables. If we are
difficulties. Changes are typically introduced for the country as a whole, leaving
studying, say, the effects of moving to a new address, then our dependent
no comparison or control groups; and they are often introduced without talung
variables may include emotional effects, changed journey to work, shopping
base-line measures before the start, making comparisons even harder. Some
habits, friends and visiting behaviour, and leisure activities, each of which has a
years ago a national education authority decided to introduce a moral education
different set of determinants. This would mean that we would need to match on
programme in its schools. A curriculum was prepared, textbooks were written different criteria for each type of effect in turn! In practice, matching is usually
and teachers were given special training courses. Shortly before the introduc- only possible on four or five variables together, and within fairly crude limits, so
tion of the new programme in the schools it was realized that the conventional that investigators tend to choose matching attributes with the widest relevance
type of school examination would be difficult to apply, since moral education is
to the different types of effects to be studied. Inevitably, some variance remains
less a matter of knowledge and more a matter of attitudes. Hurriedly some social uncontrolled, for we remain victims of the matching paradox: if we knew
psychologists were commissioned to produce a set of attitude scales to be used beforehand on which variables we needed to match, we would not need to do the
in assessment, but this took time, and meanwhile the schools had begun
research!
teaching the new programme. This made it impossible to obtain base-line
In large-scale studies it is sometimes possible to conduct a preliminary
measures, though some 'half-way' measures could have been taken. In the
enumeration survey, to collect a subject pool for matching purposes. While this is
event, a full attitude measurement programme was only undertaken at the end
costly in terms of subjects, it enables us to find many more suitable matches,
of the school year. The social scientists were disappointed because, having taken
which in turn adds to the precision of the results.
part in a 'natural experiment', they had no 'before' measures with which to
When we are dealing with larger groups or whole towns or villages (for
compare their 'after' measures. The government and the education authorities example, to study the impact of a new by-pass road), the best we can probably
were, however, delighted with the results since they showed the nation's youth do is to find another town or village as similar as possible but some distance
to have high moral attitudes - which were mistakenly attributed to the new
away, and use t h s as a control (or rather, comparison) sample.
teaching programme!
One final point to bear in mind about matching is that the implications of case
losses from either sample are quite serious: for each lost case, we also lose its
Matched-samples designs match.
We have left to last what is perhaps the most difficult aspect of before-and-after
designs, namely, the problem of obtaining two identical groups, an experimental Longitudinal survey designs
and a control group. Let us remind ourselves of the requirements for a control The before-and-after designs discussed above are really part of the larger family
group: thefunction of a control group is to show what would have happened to the experimental
of 'longitudinal' designs, so named to contrast them with cross-sectionaldesigns.
group if it had not been subjected to the experimental variable. In principle, therefore, the In cross-sectional designs we compare unrelated groups and we take all our
two groups should be as similar as possible, but what exactly does 'similarJmean
measures at more or less the same time. In longitudinal designs we take repeated
here, and how is this objective to be achieved?
measures of fhe same respondents at several time intervals.
Sometimes it is possible to draw both samples in exactly the same way from
Longitudinal designs tend to be somewhat weaker in terms of causal
a larger population, such as two probability samples (see Chapter 3). Another attribution. This is because typically there may be months, or even years,
method is to split one sample into two by random allocation to each of the two between the time the base-line measures are taken and the final measurement
conditions. These two techniques should equalize the two samples for any stage. During such a lengthy interval many intervening variables may influence
relevant pre-existing attribute or determinant, such as the volunteer error or the the effects we are studying. Also, longitudinal surveys often lack control
self-selection error. At other times all we can do is to equalize the two groups in their
samples, and they tend to suffer from case losses and consequent biases.
overall distributions on a few relevant variables, for instance by making sure
34 Questionnaire Design, interviewing and Attitude Measurement Analytic survey designs 35
Longitudinal studies may be prospective or retrospective. Examples of typical Choosing the appropriate design
prospective longitudinal studies are long-term follow-up studies of groups of
children or of major social interventions; or the study of the long-term effects of First of all, the decision must be made as to whether an analytic or a descriptive
advertising or of other communication campaigns. In retrospective longitudinal study will be undertaken (descriptive designs are the subject of Chapter 3). A
studies there may be only one data collection point, for example when studying great deal follows from this choice, and it is often risky to try to combine
a group of alumni, or there may be several. Sometimes respondents are asked to elements of both - especially if the data have already been collected! For an
furnish base-line data about themselves, as far as that is possible, for example analytic study, two issues must be carefully considered before starting:
about pub-going behaviour before and long after becoming unemployed. In
some longitudinal studies the results are fed back to the intervention agency (for I. How much is already known about the main causal variables and processes
example, the hospital, the educational institution, the market research office) as in the chosen area of enquirv?
soon as possible at every stage of the follow-up so that remedial action may be 2. How much control will there be over events; for example, the timing of the
taken. This may help to make the intervention more effective, but it introduction of an experimental variable; the allocation of subjects to groups
compromises the causal interpretation of the longitudinal design. or conditions?
In longitudinal surveys case losses become a serious problem, since they will
almost certainly introduce a bias in the results. This is a major hazard in By dichotomizing each of these issues we arrive at four groups:
retrospective follow-up studies; that is, situations where the follow-up was not
planned as part of the initial intervention but is instituted years later. If we decide A. Little is known, and the researcher has no control over events.
this month, say, to undertake a long-term reverse follow-up study of all those B. A well-researched domain, but the researcher has no power to influence
who graduated in nursing from college Z twenty-five years ago then we might, events.
with some difficulty, obtain a complete list of names, addresses at the time, ages, C. Little is known, but the researcher has power to control events.
college grades and anything else on record. But if we now send out a follow-up D. A well-researched domain, and the researcher can control events.
questionnaire to those addresses we can still find, who will be most likely to
reply? More men than women because women often change their name when There are appropriate designs for each of these categories (see Table 2.1).
they marry; those who are still active in the profession; those who have not gone
abroad or moved to a succession of different addresses; and those who think of
Table 2.1. Survey designs for analytic studies
themselves as 'successful' or who have positive memories about their college
years . . .
Little is known Well-researched domain
Panel studies
No control over events Cross-sectional designs Factorial designs
A panel study is a particular kind of longitudinal study which involves a set of Natural experiments Multivariate analyses
base measures followed by a succession of follow-up interviews or other Retrospective follow-up including multiple
measures. For example, this is how a social trend, or the impact of a political Panel studies regression
campaign, can be evaluated. Panel studies do not aim at great precision and
frequently do not have control groups. More often their function is to throw Power to control events Planned prospective follow- Before-and-after designs
light on processes of slow, informal influence and change or to illustrate the up with control sample (matched groups)
changes through which people go in adapting themselves to a new variable in Effects and intervention
their lives. studies
Inevitablv vanel studies suffer from the twin problems of volunteer bias and
4 A
contracting sample size. As each interviewing wave follows the next, it becomes
more and more difficult to locate all the panel members and to maintain their LITTLE IS KNOWN; RESEARCHER HAS N O CONTROL OVER EVENTS
interest in the study; replacements are only a partial answer to this. Another risk No causal inferences possible, only associations. Here we find the simple cross-
is that panel members may begin to take a special interest in the problems at sectional design studies. In this category are also the so-called natural experiments, in
issue; they become more knowledgeable and more critical and, to that extent,
which the 'effects' of a newly introduced variable are confounded with pre-
less representative. O r members of a panel may change their behaviour as a existing and (self)-selective factors; the retrospective long-term follow-up study
result of the greater awareness produced by repeated questioning. Neverthe- without a control group and suffering from biases due to case losses and pre-
less, successful panel studies can make contributions to knowledge which are existing factors; and panel studies, which can throw light on change processes over
obtainable in no other way.
36 Questionnaire Design, Interviewing and Attitude Measurement Analytic sumey designs 37
time -though not in causal terms -but are likely to suffer from case losses and Selected readings
volunteer behaviour .
Mitchell, Mark and Jolley, Janina, 1988, Research Design Explained, Holt Rinehart and
Winston, New York.
A WELL-UNDERSTOOD DOMAIN, RESEARCHER HAS N O POWER OVER EVENTS A practical and clear undergraduate text.
Here we can hope to formulate precise hypotheses and distinguish between the Kish, Leslie, 1987, SiatisficalDesign for Research, Wiley, Chichester, Sussex.
dependent variable and the independent, controlled and uncontrolled variables. Highlights the statistical considerations that enter into the research design process.
No causal inferences possible, but designs featuring complex multivariate
analyses (for example, multiple regression, analysis of variance) and factorial Bernstein, I.N., 1976, Validity Issues in Eualuafiue Research, Sage, London.
designs all help to disaggregate the variance. Associations no proof of causality, Himmelweit, Hilde T., Humphreys, Patrick, Jaeger, Marianne and Katz, Michael, 1981,
and direction and sequence of causality often remain uncertain because designs How Voters Decide, Academic Press, New York.
basically cross-sectional. An example of a complex series of causal research designs.
Blalock Jr, H.M. (ed.), 1985, Causal Models in the Social Sciences, Aldine, New York.
LITTLEIS KNOWN IN THE AREA; RESEARCHER CAN CONTROL EVENTS For advanced students.
Campbell, D.T. and Stanley, J.C., 1966, Experimenfal and Quasi-Experimenfal Designs for
Here is the planned, long-term, prospective follow-up of two or more near-identical Research, Rand McNally, Chicago.
samples (one of which acts as the control sample)to which individuals have been A famous and still relevant early text, concerned with evaluation studies and social
randomly assigned. Nature and timing of the intervention controlled by the experiments.
researcher. Causal inferences possible.
Random allocation hard to achieve, but is essential in order to control for pre- David, James, A., The Logic of Causal Order, Quantitative Applications in the Social Sciences
existing factors. The longer the time period, the harder it becomes to exclude No. 55, Sage, London.
influences which might affect the causal process between experimental and
dependent variable: causal identification becomes a problem, as does bias due to ONLONGITUDINAL STUDIES
case loss. Basically an extended before-and-after design with control group is needed.
Such designs often vague, not sure of their objectives; few clear, measurable Goldstein, Harvey, 1979, The Design and Analysis of Longifudinal Sfudies, Academic Press,
New York.
hypotheses.
The best way of ensuring this is by means of a completely random sampling advantage that the addresses will be 'clustered' geographically, thus greatly
method. Randomness, in this context, does not mean some sort of arbitrary reducing interviewing costs and travel expenses. (In reality, the sampling
process. It is a statistically defined procedure that requires a table or set of procedure is more complex, because we have to take account of the relative sizes
random numbers which can be generated by a computer, or can be found at the of each unit within strata and of the intended sampling fraction, but here we are
back of most statistics textbooks. If we are dealing with a small population -for concerned only with the basic rationale.)
example a register of a few thousand members of a particular association -then There are various other methods by which we can draw a representative
this might become a desk-top operation. We can number the names on the sample, though these may not meet the exacting standards of a probability
register from 1to, say, 7,500 and then use a random numbers table to select a sample. One of these is called the quota sampling method. A 'quota' in this context
random sample of, for instance, 300 names for our survey sample. The division is a cell within an overall sample, designed to have the same sociodemographic
of 300 into 7,500 is called the sampling fraction. In this case the sampling fraction characteristics as its population; that is, the sample should have the same sex
is twenty-five; that is, a random selection of one name in twenty-five. However, ratio as the population, the same ethnic and socioeconomic status groupings, the
as we shall see, this does not imply that we can run our finger down the list of same proportions of married, single, divorced and widowed members and so on.
names and choose every twenty-fifth name. The selection must be truly (In practice, fieldwork agencies tend to use rather fewer sampling variables.) A
random. Such a sample would be called a simple random sample and should be 'cell' might then be allocated to a particular interviewer or group of interviewers.
distinguished from a cluster sample (see below). Instead of being given a list of names and addresses (for example, from an
To carry out such an operation for a population of millions, or for a whole electoral register) the interviewers are given a quota: they must find and
nation, would be extremely cumbersome and expensive. There are also practical interview, say, twenty-five single, white working-class women aged thirty-six
considerations related to the geographical location of our respondents. Even to forty-five. When all the cells, or quotas, are put together they will form a
drawing a small sample of 300 names from a population of 7,500 might yield sample which should have the same sociodemographic characteristics as the
addresses that are spread far and wide and might greatly increase the cost of population, and the sample's responses to questions about a forthcoming
interviewing or might require a postal questionnaire. Even if we succeeded in election, TV programmes, frozen foods or children's toys are expected to be a
drawing a truly random sample of, say, 1,600 names and addresses from a faithful reflection of the answers that would have been given by the population
nationwide population of many millions, the geographical spread would increase as a whole. In principle this quota sampling procedure is applicable also to any
the interviewing costs to prohibitive levels. other defined population, for example divorced non-white Methodist home-
We might therefore resort to a modified sampling method which is called the owners, just so long as we have accurate population statistics (probably from the
cluster sampling method. This takes advantage of the fact that most populations latest census) to enable us to draw up our sampling quotas.
are structured in some way, or could be divided into sub-sections according to The main advantages of the quota sampling procedure are speed and low
certain characteristics; that is, they can be clustered. Thus, a large national costs. The interviewers have to use their initiative and local knowledge in order
labour union will have numerous local branches which, in their turn, are to fill their quotas and must tactfully decline to interview people whom they
grouped by state, province or county; a national electorate will be subdivided approach but who turn out to be outside the quota; that is, who turn out not to
into local constituencies, which fall into regions, which form parts of a state or be divorced, not to be home-owners, not to be Methodist and so on. The L
province and so on. The small cluster of addresses that share the same postal or procedure has often been criticized because interviewers tend to have their
zip code are part of a nation-wide mail distribution system which is hierarchically favourite locations (a bus or railway station forecourt, a shopping mall),
structured according to zones, sectors, districts, regions and provinces. Cluster favourite times (week-day late afternoons, week-end early evenings) and their
sampling is designed to make use of these structured hierarchies. As we have own expectations of what makes 'a good respondent' on sight - any of whch
seen, random sampling requires that each member of the population should may introduce biases. (Such biases may, however, be purely local within-cell
have an equal chance of being chosen. In cluster sampling this principle is applied effects, and the quota stratification is expected to minimize such sources of
to each stratum in turn. Suppose we have a population which is stratified into error.) Interviewers may also, after a long and disappointing day on the streets,
three levels: local section, region and state; and suppose further that there are be tempted to 'stretch' a category, for example by including a respondent whose
thirteen states each containing from 25 to 40 regions which, in turn, might each age falls just outside their assigned quota. Yet detailed comparative research has
contain 120 to 260 sections. What we have to do is to work systematically from shown that the results from quota samples are often closely comparable to those
level to level with our table of random numbers. First we choose, say, four of the from other kinds of representative samples. Certainly the method is widely
thirteen states at random; from each of these four states we choose, say, six used, especially in situations where speed and costs are important.
regions at random; from each of these twenty-four regions we choose, say, eight The costs of an interview survey are strongly influenced by contacting time, the
sections at random; and finally, from each of these 192 sections we choose ten time an interviewer has to spend in order to find a respondent to interview. It
people at random for our sample. This procedure would give us a sample of just has been estimated for the United Kingdom that achieving a 70 per cent
under 2,000 names and addresses which, in theory, is as accurately representa- successful interview response rate in a probability sample requires an average
tive of the population as a completely random selection would be - with the contacting time of about forty-five minutes, and this time increases rapidly if a
42 Questionnaire Design, Interviewing and Attitude Measurement Descriptiae survey designs 43
population? One approach that has been used is the snowballing technique, where
higher response rate has to be achieved. O n average, this means that an a few appropriate individuals are located and then asked for the names and
interviewer using a probability sample will successfully complete fewer than five
addresses of others who might also fit the sampling requirements. If this
thirty-minute interviews per day, compared to nine interviews per day and an
approach is repeated a number of times it might yield substantial numbers, but
average contacting time of less than ten minutes using a quota sample, and an
it is difficult to know how accurately these represent the population of concern.
even higher completion rate for telephone interviews.
For preliminary investigations and for some parts of the pilot work,
The importance of a relevant and accurate sampling frame cannot be
researchers sometimes draw a judgement sample. This description really means
overstated. We often read impressive reports in newspapers or magazines who
that accurate parameters for the population are lacking but that the investiga-
have polled their readers by means of a questionnaire printed in their own pages. tors have done their best to obtain as wide a spread of individuals as possible.
Since virtually nothing is known about the target population (other than,
There is the risk, however, that the sample will represent only a particular sector
perhaps, some kind of readership 'profile'), and since the returns are likely to be
of the population and that only very roughly. For example, if we interview a
highly biased and their representativeness unknown, no valid conclusions can be
judgement sample of the homeless, of the elderly demented or of drug addicts,
drawn from the results, either about the readers of that newspaper or magazine
then these are probably individuals who for one reason or another have come to
or about the nation as a whole, no matter how many questionnaires have been
the notice of the authorities or of certain charitable organizations, or who have
returned.
sought help or been made to seek help by their families or friends. It would be
When we are drawing a probability sample from a complete population list, for
very misleading to generalize from these cases to the wider populations of drug
example members of a trade union or a list of subscribers to a periodical, and if
addicted, demented or homeless people. Indeed, those who come forward for
we have a known sampling fraction, it is always tempting to run down the list
help or treatment are likely to be quite atypical.
and choose every Nth name. The dangers of this approach are illustrated by the
Since these kinds of samples do not meet the requirements of probability
following anecdote. A sociologist was conducting a study of pharmacists. He was
sampling, statistical assumptions about sampling errors and estimates of
interested in their urbanlrural spread, their training and social background, the
population parameters do not apply.
extent to which sons followed in their fathers' footsteps, their ethical codes and
so on. He designed and piloted a postal questionnaire to be sent out to a
representative sample of pharmacists. He obtained the most recent annual
register of pharmacists, a bound and printed volume. By coincidence, he found Sample size and sampling error
that his sampling fraction corresponded exactly to the number of names on each
The field of sampling is in some ways highly technical and full of statistical
page, so he sent a questionnaire to the first and last names on every page of the
register. Only when it was too late did he realize his mistake. Since the list was
it falls for the beginner. Often it requires compromises between theoretical
sampling requirements and practical limitations such as time and costs.
in alphabetical order according to surname, the first surname on each page was
Consider, for instance, the question of sample size. Common sense suggests that
often the same as the last surname on the preceding page. Not only had he,
a larger probability sample will give a better estimate of population parameters
therefore, unwittingly sampled by pairs, he had also thus increased ('over-
than a smaller one, but will also be more costly. Yet the size of a sample is, of
sampled') the probability of finding fatherlson pairs in his sample. And if, to
itself, not very important: a national sample of many thousands might still be
avoid the consequences of his error, he had arbitrarily discarded one member
very poor - if, for example, it consisted solely of readers of a particular
of each pair, he would have 'under-sampled' for the prevalence of fatherlson
newspaper who had returned a questionnaire. A sample's accuracy is more
pairs.
important than its size. A properly drawn sample of fewer than 2,000 adults can
So we note that the only safe way to draw a probability sample from an
give us more reliable estimates on a population of many millons, for election
alphabetical list is by simple random sampling with the aid of a table of random
polls or other types of social research, than a huge sample of a quarter of a
numbers. Even if the names are not in alphabetical order it is wiser to use a
million which is poorly drawn. Statistical tables exist which will show the degree
random numbers table to avoid any unknown bias.
of precision (the sampling error) which is theoretically obtainable for samples of
different size; thus, the accuracy of a given percentage result might be plus-or-
minus 2 per cent for a sample of 2,000 or 6.5 per cent for a sample of 200. A
'Snowballing' and judgement samples finding that 30 per cent of our sample said 'yes' to a particular question must
therefore be interpreted to mean that the true figure for the population will
As we have seen, problems arise when we wish to sample a population with
most likely lie between 28 per cent and 32 per cent, or between 23.5 per cent and
unknown characteristics. For example, how would we obtain a sampling frame
36.5 per cent, depending on sample size, with important implications for the
for football supporters, amateur saxophonists or homosexuals? Even if we could statistical significance of group differences. These figures are optimum theoret-
operationally define such people, how would we sample them in a way that ical estimates, based on the somewhat unrealistic assumptions that we have an
permits generalizations? And how would we know whether, or to what degree,
accurate and up-to-date sampling frame, that we have conducted the sampling
our sample is representative, since we have no accurate parameters for the
44 Questionnaire Design, Interviewing and Attitude Measurement Descriptive survey designs 45
operation faultlessly (or have been able to compensate for any biases), that the certain age, subscribers to X magazine, people who regularly ride a bicycle to
fieldwork has been error-free, that there is no non-response and so on. work?
Obviously, the true sampling error will in practice be larger and will depend If some population parameters or an accurate sampling frame are available,
greatly on any deviations from the ideal sampling procedures. what method should be used to draw a sample that is truly representative? The
But sampling error is only one of several reasons for choosing a given sample simple random or equal probability sample is often regarded as ideal, but may be too
size. Perhaps more important is the number of sub-groups we wish to compare. costly or impractical. Various kinds of cluster random samples may be possible, and
If there are only two, or three (say, two sexes, or three age groups), then having offer savings, but may increase the sampling error. Quota sampling is frequently
some 200-300 respondents in each might suffice. But if we have many sub- used where constraints of time and costs apply; the quaLty of a quota sample will
groups (for example twenty-three areas, or firms, to be compared with each depend on the number of attributes used for clustering the interviews which, in
other individually),then a much larger sample will be needed to reduce error and turn, depend on the sampling frame. Sampling from lists should be done with the
to attain statistical significance. Another important determinant of prospective aid of a table of random numbers.
sample size will be the nature of the dependent variable. If this is an interval scale What about sample size? This will depend, first, on statistical considerations
(see Chapter 9), then we can hope to deal with group averages in our such as the accuracy of the estimates required and the problems of statistical
comparisons and our sample need not be larger than a few hundred. But if the significance; second, on the proposed comparisons between sub-groups: the
dependent variable is nominal or categorical, for example sixteen different types more sub-groups, the larger the sample needs to be; third, on the complexity of
of crime, or a dozen consumer behaviour patterns, then reaching statistical the dependent variable(s); and fourth, on time and resources.
significance would probably require over a thousand respondents. If we have Finally, a few words of warning. The achieved sample will be less accurate than
more than one dependent, variable, as might easily happen if we have an analytic the designed sample. What steps should be taken to anticipate and reduce non-
design, then this further compounds the problem. response bias in the results, especially in mail surveys (see page 106)? People
In short, sample size will be determined by theoretical requirements (such as sometimes draw predictions from descriptive studies, for example public opinion
sampling error, cluster size, required accuracy of population estimates), by the polls at election time. How may over-interpretation of the findings be avoided?
precision of the sampling operation, by the number of sub-group comparisons For a 'trends' study using repeated probability samples (for example for market
we aim to make and by the nature of the dependent variable - and ultimately, research purposes), each successive sample will need to be as similar as possible
by constraints of time and costs. to the first achieved sample. It is not sufficient to repeat the sampling process as
before. Any sampling procedure should be tried out for feasibility and
completeness in a pilot exercise.
Choosing a survey design for a descriptive enquiry
The first question to ask is: is it possible to study a complete population (for
example, all the employees of a particular firm) or will a sample have to be Selected readings
drawn? For a complete population study, defining that population may be a
problem: where to draw the line? Case losses or non-response owing to absence, Dillman, D.A., 1978, Mail and Telephone Sumeys, Wiley, Chichester.
illness, refusal etc. are often a problem because they may cause biases. Refusals Hoinville, Gerald and Jowell, Roger, 1978, Sumey Research Practice, Heinemann, London.
should be counted and studied separately from non-contacts (using whatever Straightforward and practical general text on survey methods. Especially good on
interviewer observations or background data are to hand). Where possible, organizing fieldwork and postal surveys.
biases should be countered by statistical weighting. Successive population
censuses may reveal changes, but no valid causal interpretations are possible.
For a sample study, the question is: should it be representative; that is, should ONSAMPLING
it be possible to obtain population estimates from the sample? O r should it aim
Rossi, P.H., Wright, J.D. and Anderson, A.B. (eds), 1983, Handbook of Survey Research,
at representing certain special sub-groups, for example non-voters in a
Academic Press, New York.
particular election? O r some kind of mixture, for example a population sample See chapters 2 and 5 on sampling.
in which groups of special interest have been deliberately oversampled, such as
families with twins? Henry, Gary T., 1990, Practical Sampling, Applied Social Research Methods Series No. 21,
For a representative sample, is there access to any kind of sampling frame? Are Sage, London.
there records, address lists or data banks from which a representative sample A practical guide to the sampling of both general and special populations.
may be drawn? A sample cannot be drawn if almost nothing is known about the Moser, C.A. and Kalton G., 1972, Survey Mefhods in Social Invesiigafion, Heinemann,
population, for example women who have had any kind of miscarriage. Can London.
population parameters be operationally defined, for example abovelbelow a See chapters 4-8 on sampling theory and methods.
46 Questionnaire Design, Interviewing and Atfitude Measurement
Kalton, Graham, 1983. Introdudion to Suruey Sampling, Sage University Paper No. 35, Sage,
London.
A clear introduction to probability sampling.
Kisch, L., 1965, Survey Sampling, Wiley, New York.
For the advanced student.
PILOTWORK
Cochran, W.G., 1977, Sampling Techniques, Wiley, London.
Hess, Irene, 1985, Samplingfor Social Research Surveys 1 9 4 7- 198 0, Survey Research Center,
Institute for Social Research, University of Michigan, Ann Arbor, Mich.
Detailed account of ISR sampling practices.
ONVALIDATION
Belson, William A., 1986, Validity in Survey Research, Gower, Aldershot, Hants.
Questionnaires do not emerge fully-fledged; they have to be created or adapted,
Brinberg, David and McGrath, Joseph E., 1985, Validify and the Research Process, Sage, fashioned and developed to maturity after many abortive test flights. In fact,
London.
every aspect of a survey has to be tried out beforehand to make sure that it
works as intended.
It might be thought that once the main decisions about the design of the
proposed research have been taken the researcher would soon be able to make
a start with question writing and data collection. This, however, is very unlikely.
If we think of any survey as having to pass through a number of stages -from
the initial formulation of basic ideas to the specification of the research design,
followed by the fieldwork, then the data processing and statistical analysis and
so on to the writing of the final report (see Chapter 1)- then we must allow a
substantial period of time for the construction, revision and refinement of the
questionnaire and any other data-collection techniques. Questionnaires have to
be composed and tried out, improved and then tried out again, often several
times over, until we are certain that they can do the job for which they are
needed. Sometimes we can borrow or adapt questionnaires from other
researches, but there still remains the task of making quite sure that these will
'work' with our population and will yield the data we require. This whole lengthy
process of designing and trying out questions and procedures is usually referred
to as pilot work.
Each survey research presents its own problems and difficulties, and expert
advice or spurious orthodoxy are no substitutes for well-organized pilot work.
Piloting can help us not only with the wording of questions but also with
procedural matters such as the design of a letter of introduction (and from
whom it should come), the ordering of question sequences and the reduction of
non-response rates. We should realize from the beginning that pilot work is
expensive and time-consuming, but avoiding or skimping on pilot work is likely
to prove more costly still.
The length of time taken by the pilot work can differ greatly, and should not
be underestimated. If an enquiry follows a well-known path, using adaptations
of tried-and-tested instruments, a few weeks of pilot work may be sufficient for
an experienced team, though even then short-cuts may cause fatal errors. It
should not, for example, be assumed that an American or Australian question-
naire will work equally well in Britain, or vice versa, nor even that a
48 Questionnaire Design, Interviewing and Attitude Measurement Pilot work 49
a
your final report will need to contain a table (or perhaps series of tables) to
to start with an item 'pool' of perhaps as many as a hundred items or more, to
be administered to a substantial sample (say, 250 respondents or more), in order
show how true or partly true this is. Presumably such tables will have headings that the 'best' items can be selected by statistical means (see Chapter 9). It is
across the top such as 'older brother, non-smoker', 'older brother, light smokerf1 possible to administer such item pools to the main fieldwork sample and to
'older brother, heavy smoker' and 'no older brothers'; and the same for older conduct the scaling procedures afterwards, but this might overburden respond-
sisters; and some headings for those who have more than one older sibling of ents with a survey instrument of prohibitive length. It is therefore preferable,
one or both sexes. This could get quite complicated if we sub-divide the sample and less wasteful, to conduct any scaling procedures in advance of the main
further according to whether any such older role models live at home or not; and fieldwork on separate pilot samples. In this way, the length of each scale may be
what the age difference is between each older sibling and the respondent. reduced from an item pool of many dozens to a short and more effective scale of
Presumably we shall want to do this analysis separately for boys and for girls, perhaps fifteen or twenty items. When finalised, such scales can then be included
perhaps divided into age groups as well. And what would the entries in such in the main fieldwork without making the questionnaire too long.
tables be? The respondents' claimed number of cigarettes smoked per week? This principle of separation can often be applied quite generally. The mid-
Smoking expenditure figures? Over what period? It can be very useful to phase of the pilot work, after all possible questions have been generated, is likely
construct such dummy tables long before the fieldwork is started; they will to have produced a questionnaire that is much too long. It is therefore good
highlight the need for special questions (for example, the age interval between practice to divide the items up into separate booklets that are relatively short and
the respondent and each older brother or sister, if any); the need for summary contain roughly similar items which can be tried out on separate pilot samples.
statistics (for example, expenditure averages over certain periods); and probably Thus, one booklet might contain 'factual' questions on the respondent's
the need for a larger or differently chosen sample for the exploration of this behaviour (say, with respect to toothbrushing in a dental health study or video-
particular hypothesis - or the decision to give up the whole idea! tape borrowing in a mass media survey); another might contain 'awareness'
Altogether, as we saw in Chapter 1, it is often very useful to run through the questions; a third might be concerned with opinions and attitudes (including
sequence of a survey enquiry backwards as part of the pilot work. In order to draw scales where relevant); a fourth booklet might ask about demographic aspects of
conclusions about certain hypotheses, the relevant tabulations must be availa- the household (ages of children, size of the home, job history, education and so
ble, which will depend on certain questions or scales or other measures having on). Each of these booklets will follow its own journey of pilot development,
been given to samples that are precisely relevant to this enquiry. If questions from the initial 'drawing board' phase, through several improved and shorter
have been left out or forgotten, if the answer categories do not bear on the versions, to the final text which will be included in the main fieldwork.
hypotheses, or if the sample is inappropriate to the issues, the statistical analysis An added advantage of having such 'sub-assemblies' is that it will break up the
will be meaningless, and the intended hypotheses cannot be examined. From later phases of the pilot work into a number of smaller operations without
start to finish every stage of a survey must be linked to what is to follow and to overburdening the respondents. It will also give the researcher valuable
what has gone before. The whole set of operations must be coherent and experience in the relevant administrative proceudres: drawing small samples,
integrated, and a reverse pilot exercise will show up any weaknesses. It must contacting respondents, explaining the purpose of the survey, timing each
always be possible to justify the inclusion of every item; it is simply not good operation and so forth. When ready, several subsections may be piloted together
enough to say that 'we shall see what to do with the responses later' or to include (perhaps experimenting with different question sequences), and eventually we
a number of items just for luck - these are signs of sloppy design and will need to pilot the questionnaire as a whole.
inadequate pilot work and will most probably lead to wasted effort and
misleading results.
Exploratory pilot work
Organization of pilot work The earliest stages of the pilot work are likely to be exploratory, and will be
primarily concerned with the concepfualization of the research problem (see
Since almost every aspect of a survey enquiry can be made the subject of pilot Chapters 1and 5). They might involve lengthy, unstructured interviews; talks
work, a line has to be drawn somewhere and some issues will receive greater with key informants; or the accumulation of essays written around the subject
priority than others. The greatest amount of pilot work will usually be devoted of the enquiry. For example, suppose we are doing a study of attitudes to the
(a)to the measurement of the dependent variable and (b)to those problems that, -
police. After looking at ~ublishedresearch on this topic,ZiidGaring in mind- t G
at least at the outset, are expected to be the most difficult. If the survey is to Gbjectives of our studyIhow do we make a start with our questionnaires? How
contain inventories, projective techniques or attitude scales, these are likely to do people 'see' (perceive) the police? Are they seen as a threat, as enemies? Or
require separate and quite substantial pilot work (see Chapters 11~12and 13). as helpers and protectors? Or a bit of both? How do people feel about police
Yet the pilot work should also be seen as an effort at condensation of the main uniforms, and are these feelings linked to attitudes to other uniforms (military,
fieldwork.If the survey instrument is to contain groups of items that are to form nursing, firemen, prison staff)? Are the police seen as authority figures and
an index, a scale, or an inventory which will yield a score, then one would need
52 Questionnaire Design, Interviewing and Attitude Measurement Pilot work 53
perhaps subconsciously linked to parental authority? How do people form their wording of the question and the design of the answer-categories - and these
attitudes to the police: from the media? from experience of traffic police? from two problems interact: if the respondent knows the answer categories (either by
having been the victim of a crime? Do people believe the police to be biased reading them on a self-completion questionnaire or on a show card, or by having
against certain minorities? Are they seen as corrupt and likely to tamper with the interviewer read them out as a 'running prompt'), this will influence the
evidence? How would a respondent feel if his or her son or daughter wanted to meaning given to the question and may constrain the response, while the
enter the police force? These are but some of the issues that might emerge interviewer, too, is influenced by the answer categories (even if they are not read
during a free-style, exploratory interview. (How to conduct such 'depth' out). By first experimenting with the question in 'open' form we can discover
interviews will be discussed in Chapter 5).A couple of dozen such wide-ranging, how respondents spontaneously interpret the question (their frame of refer-
unstructured interviews might give us the beginnings of a conceptualization of ence); we can then either modify or 'sharpen' the question in accordance with
the problem, and this will enable us to develop what is sometimes called a 'hidden our purpose, or 'guide' the respondent by offering additional answer categories
agenda'. which the respondent might not have considered. We might also include
A hidden agenda is a flexible list of topics (not actual questions) to be covered in categories dealing with popular misconceptions that we might wish to explore.
an exploratory interview. It will be shared among the several depth interviewers Suppose, for example, that we are interested in the ways in which people save
concerned, but it is not expected that each interview will cover all topics nor will money. A wide-open question about savings will yield a variety of responses
topics always be approached in the same order. Whenever a topic proves itself to which could be used to form answer-categories later. However, typically some
be productive with a particular respondent, the interviewer will 'flow with the respondents will use a'pot'system (that is, setting aside small sums of money for
tide', even at the risk of missing out some topics. In the same way, new topics a special purchase) which they may not regard as Savings', while some may fail
may be added to the agenda: for example, a respondent may have a lot to say to mention mortgage repayments or insurance policy premiums because these,
spontaneously about the police in connection with drink-driving offences. too, may not be considered 'savings'. It all depends on how the term 'savings' is
Perhaps this topic had not been considered for inclusion initially, but after interpreted, but clearly, as it stands, the question may yield a considerable under-
discussion it might now be added to the hidden agenda. estimate, and we might not have realized this if we had not piloted the question
From these early, exploratory interviews (which should always be tape- in open form.
recorded, and listened to by the depth interviewer afterwards, as well as by other So what to do? It depends, first of all, on the purpose of the question. If we are
members of the team), a rough shape of the enquiry will emerge. For example. interested in the subjective meanings attached to the concept 'savings' then
it is likely that a set of questions will be needed to obtain people's direct, personal clearly we should not 'lead' respondents by suggesting additional categories to
experiences with the police - in different roles and contexts: as a speeding them - or at least not in the initial question but perhaps later, in the form of
motorist; as the parent of a young delinquent; after being burgled, living next- follow-up probes. But if the purpose of the question is to obtain as complete a
door to a police officer. Another likely set of questions will deal with mass-media picture as possible of any forms of savings behaviour (the scope of which will
preferences, with special attention to crime serials, law-and-order issues, road- have to be carefully defined in the planning of the research), then we can either
safety items and so on. Drink-driving issues may also require a special set of sharpen the question by offering a definition which includes mortgage
questions. And there will probably be a need for several attitude scales, the repayments, insurance premiums, etc., or we can add these to the list of answer
dimensions of which can already be sketched, even at this stage. categories - or perhaps we may do both. It will depend on subsequent pilot
Once this exploratory stage has given us a 'feel' for the problem we are work with the revised question, trying out the question in several different
investigating, the rest of the pilot word can proceed in an organized form, not forms and comparing its 'yields' in parallel pilot samples. (See page 61.)
unlike an assembly line. Within the overall plan certain sections may be How do we know when a question needs rewording? In principle, we always
developed further, perhaps by particular team members, piloted as separate try to see whether the responses tell us what we need to know but, in addition,
booklets and eventually put together as a coherent whole. there is always the problem of inadvertent biases. A poor question will produce
a narrow range of responses or will be misunderstood by part of our sample; the
question may be too vague, or it may ask for information which the respondent
Piloting factual questions does not have or cannot remember; it may be a leading question ('Do you
generally refrain from dropping litter in the street?') which biases the answers,
A useful way of malang a start with question wording (see Chapter 8) is to or a question that operates at the wrong level of intimacy for its purpose. A
consider some of the factual questions that will be needed. These often take the question may be too wide or too narrow in scope; it may require an introductory
form of multiple-choice questions (see Chapter 7).As we shall see, all multiple-choice sentence or a supplementary question (a 'probe'); it may be too colloquial or too
questions should start their lives as open-ended ones. We should never just 'jot technical, too intimate or too abstract, too long or too short. As we have seen
down a few questions', invent some answer categories and then insert them in (see p. 49 above) some questions are well understood and seem perfectly
the later stages of the pilot work without first piloting them in 'open' form. reasonable but produce answers that cannot be classified.
When we design multiple-choice questions we have two problems: the If they concern variables that are of special importance, some questions may
Pilot work 55
54 Questionnaire Design, Interviewing and AffifudeMeasurement
go through as many as eight or more revisions before producing satisfactory
results, and it is not uncommon to 'use up' several hundred respondents in pilot
work. Even questions that are 'borrowed' from other surveys need to be piloted
to ensure that they will work as required with our kind of respondents. This is
particularly the case with surveys of school children and with questions that
have been translated from other languages.
By the time a self-completion question is in its final form it may have
undergone many changes since it was first designed. On page 54, for example,
is question 18 from a recent British national census.
There was room on the census form for responses concerning six members of
the household, so the answer categorieswere repeated six times across the page,
each preceded by a question about the address of the person's workplace. Note
that this was a question about the journey fo work, not about the journey home
or elsewhere.
It is obvious that this question must have been piloted many times and has
undergone some changes since it was used in the preceding census. The
question appeared in a block of questions about the respondent's current or most
recent paid job within the last ten years. Its form must have been decided to meet
one or more specific purposes; for example, if the aim of the question had been
to ascertain the costs or the duration of the journey to work, then the wording of
the question would have been quite different. The question is preceded by a
heading, which is often a useful way to focus the respondent's mind on the issue,
and this is followed by several definitions and guide-lines. Thus, the phrase 'the
longest part of the daily journey' was clearly understood to mean either time, or
distance, by respondents in the pilot sample; to make sure that all respondents
understood the question in more-or-less the same way, a qualifying clause was
added: '. . . the longest part, by distance, of the daily journey. . .'In the pilot work
it must also have emerged that some people go to work by different means of
transport on different days, hence the guide-line 'show the means most often
used'.
Category 5, 'Driving a car or van', must have seemed simple enough; yet
evidently people driving three-wheeled vehicles or motor caravans ('campers') to
work must have been uncertain how to answer, so another guide-line was
included. We note that there are separate categories of driving a car or van, and
being a passenger in such a vehicle; this distinction must have been relevant to
the purpose of the question. We also note that several of the categories contain
more than one mode of transport, for example 'Bus, minibus or coach (public or
private)'. We must, in making such combinations, not only be guided by some
form of logic (for example two-wheelers; three-wheelers; four-wheelers; and so
on) but, most importantly, by the purpose of the question: why do we need to know
these things? How important d l it be for the users of the information to be able
to distinguish between each and every mode of transport? Against this must be
set the general desire to keep questions relatively short: we have ten answer
categories here, which is about the most that an average respondent can use or
distinguish. So for every combined answer category there must be some
rationale, some justification, based partly on pilot work and partly on the
h
d
purpose of the question.
We might t h n k that, by now, the question had exhausted all possible means
S o u m : From 1991 Census, England. Crown Copyright.
56 Questionnaire Design, Infemiewing and Attitude Measurement I Pilot work 57
ending occurs to them. It should be emphasized that at this stage we are not
and modes of transport, but we must never become complacent: almost
invariably we shall need the 'Other (please specify)' category. Not only will this trying to obtain some kind of 'true measure' of each respondent; we are still in
the pilot stage and so we are trying to obtain ideas, concepts, forms of words and
help respondents who come to work by unusual means, for example by water
material for future questions. The approach is 'projective', in the sense that we
bus or on a horse; it will also show, if such frequencies remain low, that the
principal categories cover virtually the entire domain of answers and that the are trying to get respondents to reveal something of their own thought
processes
-- and feelings.
pilot work has been effective. Should, however, a substantial number of
respondents make use of this category, then these responses will have to be The way to handle these responses, obtained from small pilot groups, is to
take each item in turn and then to list or type out all the responses to each item.
classified and coded as part of the data-processing stage (see Chapter 14),using
For example, here are some responses to the pilot item A YOUNG WOMAN
additional categories, so not all is lost.
WHO DRINKS TOO MUCH. . .
And finally, what about the respondents who do not have a journey to work
at all? For them, there is a special category: 'Works mainly at home.' Now we
may hope that every paid job holder in the country will be able to answer this Male respondents:
question but, until all the results have been processed, we cannot be sure. That is nice to have a date with if she pays
is one reason why the Office of Population Censuses and Surveys conducts is an easy lay
what are called 'post-enumeration surveys' (see Selected Readings). They send is no worse than a young man who drinks too much
out interviewers to hundreds of addresses from which census forms have been is trying too hard to mix in
received to conduct meticulous quality checks. In respect of a very similar shouldn't
journey-to-work question used in the 1981 census they found more than 8 per is probably oversexed, frustrated, or having a period
- unhappy, I think
cent disagreement between what was stated on the form, and what turned out
to be the case at the interview. A third of these disagreements were due to isn't really my- type
--
mistakes by the form filler, 43 per cent were due to misunderstandings of the occasionally is normal; frequently is neurotic
instructions, 7 per cent were because the respondent's personal travel circum- usually ends up wishing she hadn't
stances were not catered for on the census form and the remainder were due to is undesirable
other causes. Thus, despite extensive pilot work, the question did not always is easv
J
work as intended, with consequent error-variance in the final results. is worse than a young man who does, by far
is OK
must be careful of the company she does it in.
Exploring percepts and attitudes Female respondents:
can end up being sick
Questions of this kind call for a subtler approach. Here we can make use of the is sad
sentence completion technique (see Chapter 12). Thus, in the previous example is fine, if she has control of certain situations
concerning percepts and attitudes about the police, we may proceed from pilot is likely to lose her friends' respect
interviews using deliberately vague, exploratory questions, such as 'Do you is a hateful sight
think the police can be trusted, on the whole?' to self-completion questionnaires should be helped
using open-ended sentences such as the following: is not worse than a man who does
/ is on the road downhill
Sometimes I wish the police would . . . is trying to forget something
Without the police . . . tends to be looked down on
The police are always trying to . . . might eventually lose some of her own self-respect and that of her friends
A uniformed policeman makes me feel . . . usually has a problem
When I am out alone at night . . . is perfectly entitled to if she wishes
When 1 hear a police siren I feel . . . ought to stay at home!!
People who trust the police . . .
In conceptual terms, these responses raise the issue of 'double standards': do
the same standards apply to young women and to young men? Do male
and so on. Thus it becomes possible to move from the time- and effort-
respondents apply different standards compared to female respondents? This
consuming technique of personal interviews to a written approach: sets of such
might well be an issue to be explored further, in the research as a whole.
incomplete sentences may be given to single respondents, or to groups, who can
Conceptuallyfwe also notice that some male respondents link too much drirkng
be asked to complete each sentence in writing with whatever spontaneous
58 Questionnaire Design, Inferviewing and Attitude Measurement Pilot work 59
by a young woman with sexual behaviour, whereas some female responses They have to be recorded because they are very special.
make a link with the loss of self-respect and the respect of friends (though one Because they want to know how many people are in the country.
wonders whether there is also a hdden l i d here with sexual behaviour). Should
this be followed up and form the basis of a hypothesis? But some respondents This seemingly innocuous question which looks very much like a 'knowledge1
avoid moral comment and focus on more practical issues: cost; being sick; social item, was designed to elicit the extent to which these young people had advanced
mixing. Still other respondents try to suggest causes: frustration, having a beyond the mainly concrete, personal level and could give explanations in terms
period, social striving, neurosis, trying to come to terms with a problem. Should of broader system needs. A set of multiple-choice items could readily be based on
we perhaps explore these causal attributions further? Ought we perhaps to the responses to such pilot questions, and scores could be derived for each pupil
think of making up an inventory of perceived causes of drinking too much? Or to show the level of political development reached. As before, this technique has
are these attributions merely covert forms of disapproval? O r part of a broader the added advantage that it uses the adolescents' own language and expressions.
set of stereotypes about women?
Such a set of responses is also useful in terms of question wording. There are
spontaneously used terms here which we might use in formulating questions: Piloting other aspects of the study
'an easy lay'; 'trying too hard to mix in'; 'oversexed: 'be careful of the company
she does it in'; 'a hateful sight'; 'on the road downhill'; 'trying to forget Question wording is but one of many aspects of pilot work. To refer once more
something'; 'lose her self-respect'. One could readily see how some of these to the journey-to-work question above, layout is of paramount importance,
responses could be turned into attitude statements, for example 'a young woman especially when dealing with a self-completion questionnaire. We note, looking
who drinks too much is perfectly entitled to do so' (agree/?/disagree).Such at the census question again, that all the categories have been 'justified to the
attitude statements have the added advantage that they have not been made up right', so that all the answer-ticking boxes are one above the other and on the
by the investigator (see p. 179). right-hand side of the page. We might, perhaps, have put these boxes on the left,
Another approach which is particularly useful in the early stages of pilot work and justified the categories also to the left - pilot work can show which is best
on attitude questions is the apparently simple but actually projective (see Chapter here. Nor should we only be concerned with the convenience or comprehension
12) question. Often, it can be made to look like a factual question, and the of the respondents; we also need to think of the data processors, who will have
respondent may not be aware that the response also reveals values and attitudes. to enter the data on their computers. Next to each response box there is a
For example, a study of the attitudes of psychiatric nurses used, in the early number, located in such a way as not to irritate or confuse the respondent and
stages, the question: 'Why do patients sleep together in dormitories?' To many yet making it very easy to enter (and to check) the numerically coded answers
nurses in traditional hospitals the question seemed easy and slightly absurd. using a computer keyboard - imagine the scope for errors if, say, the categories
Some gave historical explanations, others said that it was for ease of observation had been justified to the left, yet the numbers had remained where they are, on
by the night staff, but one middle-aged ward sister wrote: 'Because it is the the right!
correct way to sleep.' o n t h e actual census form, colour and contrast were used to good effect, and
Here are some responses by 13-14-year-olds who took part in a political such aspects can and should be piloted, too. Thus, the heading 'Daily journey to
socializationenquiry. The question was: 'Why do all births, deaths and marriages work' was set in white lettering on a dark-green background rectangle; the
have to be officially recorded?' guide-lines were printed in dark brown on a pale green field; and the answer
categories were not printed, as here, below the guide-lines, but alongside them,
It is something you are very happy about and something you a r y e r y sad to the right - using dark-brown lettering on a contrasting, pale-brown
about. background. These are not matters that should ever be left to the printers, or to
We must know how many people die, have married and have just been born. the artistic licence of the designers; only systematic pilot work can determine
Because of accidents, then they could contact the parents. how such aspects influence respondents and help or hinder the survey. Often
Marriages have to be recorded because of certain reasons about ages and such techniques subtly guide and clarify the respondents' task, though they may
parents' consent. not be aware of this. On the census form, to give a further example, all the
So that no-one can pretend they are someone else or say they have been answer boxes were in white, thus gently helping respondents to put their ticks
married when they aren't. in precisely the right place - some respondents can be remarkably careless in
So that the government can tell how many people have been born, married or the way they place their ticks! And they were told to tick, not to ring, underline
have died. or place a cross by the chosen response category.
So that the mother cannot change the baby's name. On another point - of detail, the frequently used 'escape' category 'Other
So that they can be shown to the family. (please specify)' must always be followed by a space, or some lines, to
A policeman might want to trace someone. accommodate the answer - and this, too, should be piloted. A large space
So the churches know how many deaths, marriages and so on. suggests that a lengthy response is expected; a small space makes the response
60 Questionnaire Design, Interviewing and Attitude Measurement Pilot work 6 1
seem unwanted. So how much space is neither too much nor too little, but just I years, for instance a household expenditure survey or a mass media survey. A
right? standard questionnaire may be evolved in which only the details are changed to
There are many other aspects of an enquiry that can and should be piloted. I correspond with the relevant day, week or month. Such a situation is sometimes
used for split ballot type of pilot work. The whole~pilot_sample
is divided up into
The actual procedures for drawing our sample have already been mentioned.
Letters of introduction should be piloted: how effective will endorsement by a
1 two or -mFm>tched sub-samples, and two or moreversions of the same
prestigious source be in securing widespread co-operation by respondents? How q-iistionnaire are produced; for instance on different-coloured paper or with a
'official' should it be? Will some people be put off by it? We should pilot various change in question order. Studying the results of such a rigorous comparison
other methods of reducing non-response in postal questionnaires. For example, will show the effects of such variations in technique, which can be utilized in
if we need replies from parents in a study of school children, should we (a) send subsequent surveys.
letters accompanied by a self-addressed envelope to the parents' homes by post;
or (b) hand each child in a classroom a sealed envelope addressed to hislher
parents, the response envelopes to be brought back to the school and collected Processing the pilot data
by the class teachers; or (c) sample only a few children in each classroom, and
have the parental returns mailed back to the research team separately by each During the pilot work we should also think well ahead towards the analysis
parent? Each method has advantages and disadvantages, and the best way to stage (see Chapters 14 and 15). For instance, suppose we are doing a survey of
decide which method to use is to conduct three parallel pilot runs. the adjustment of foreign students in this country and that we propose, as
We can pilot the effects of the age or sex or ethnicity of our interviewers on important variables for assessment, their pattern of friendship choices, their
the responses, the quality of the paper we use, the ordinal position of multiple leisure-time activities and their contacts with home. We shall of course be
responses (do people tend to pick the first and the last categories?)and the timing thinking of ways to question students on these points in some detail, but we
of each section of the questionnaire to avoid making it too long. should repeatedly ask ourselves exactly why we need these answers and what
It should be clear from the above that, if these issues are important, pilot we propose to do with them later on. For instance, with regard to friendship
studies of some size, frequency and complexity will be needed. Sometimes it is choices, do we want to obtain an estimate of the extent to which respondents
sufficient to send out three or four experienced interviewers to collect data on chiefly make friends with members of their own nationality? If so, we shall have
some fifty cases and suggest improvements. Their purpose is not the collection to ask for the nationality of each friend mentioned and compute some kind of
of findings but the testing out of questions and procedures. They must put ratio. Do we want to investigate in what ways various social contacts have
themselves in a critical frame of mind, thinking also of difficulties that less influenced their attitudes towards this country? If so, then we shall have to get
experienced interviewers might encounter. Their job is to make sure that the an evaluation of the 'satisfactorinessJof each relationship. If we ask them about
questions make sense to the respondents (including problems of translation and their leisure-time activities, perhaps we shall wish to compare these to their
the use of vernacular) and that the categories provided for multiple-choice previous leisure-time activities in their home country. Have we remembered to
questions or field coding are mutually exclusive and comprehensive. When the ask about those, and is the information in comparable form? Are we, at a later
interviewers encounter problems in the field they should be encouraged to try stage, going to want to calculate an over-all 'adjustment index' or a 'loneliness
to design better questions or alternative categories - and in this they can often quotientJ? If so, now is the time to work out exactly how this should be done,
secure the assistance of respondents. ('How do you think this question should be using the pilot data.
~ 6 i kiAd
s of thinking ahead does not necessarily mean added questions and
asked?') They should also be encouraged to be critical of layout,
instructions ('If Yes, go to Question 243, space for verbatim re ording, tTting greater complexity; sometimes it helps to simplify questions. A well-known
example concerns asking respondents for their age. This can sometimes cause a
fieldcoding issues, and of anything else that might impede the study.
In the case of small samples, all the answers to each question should be copied certain amount of embarrassment, and so we must consider what we are going
out on sheets of paper, one sheet per question. In this way one can quickly see to do with this information. Very likely we shall use it in our tabulations in
whether the question has any obvious faults and whether the answers seem to condensed form, such as 'forty-five and overJ versus 'under forty-fiveJ, or in
bear on the issues required. Sometimes it helps to group the questionnaires steps of ten or fifteen years at a time. If this is so, then we need not ask
before copying, in terms of socioeconomic status, gender, age or some other respondents for their exact age; they should merely be asked to state whether
relevant variable. Such copied-out, grouped sets of responses are of great help in they are forty-five or over, or under forty-five, or else whether they are in their
determining the sensitivity of questions and in devising coding categories or twenties, thirties, forties and so on, thus causing less embarrassment. There is
multiple-choice answer categories. little point in burdening respondents and risking reduced response rates by
Though pilot samples tend to be small, some of the anticipated group asking for detailed answers when the results are going to be treated in highly
differences (for example between people who have, or have not, been burgled) condensed form in the analysis. Each time we should ask ourselves: what is this
should emerge and can be given greater scope and precision. question doing here, and how do we propose to analyse it later?
Sometimes an organization has to repeat a set survey many times over the Due attention should be given to the statistical and data-processing tech-
62 Questionnaire Design, Inhruiewing and Attitude Measurement Pilot work 63
niques to be used in the analysis. A well-phrased question emerging from careful must you imagine that an expert could simply sit down at a desk and draft the
qualitative pilot work may produce responses that are hard to quantify. The best possible set of questions. An expert may make fewer technical errors (such
calculation of an index or scale score may produce computational and scoring as using overlapping categories; for example, age 20-25; 25-30; 30-35; etc.) and
difficulties. The introduction of probes and the production of multiple answers may have experience in specialized domains (for example, asking questions
to the same question each produce problems of quantification to which attention delinquent behaviour), but experts would be the first to use pilot work for
should be paid during the pilot stages (see Chapter 14). There is no point in their own surveys and would insist on much greater precision in the design and
asking 'interesting' questions that produce responses which cannot be analysed. purpose of any question. So it may be interesting to have an expert 'pick your
1 questions to pieces', but you willlearn more from doing the pilot work yourself,
Z d i t will produce new and better questions as well as a reformulation of
On whom should our questions be tried out? 1 question objectives. An expert should never be used as a short-cut.
show how effective the changes have been. The questionnaire or interview
schedule can now be amended and standardized for fieldwork application.
' I 'HE EXPLORATORY INTERVIEW
The benefits of pilot work
Pilot work may be costly, but it will actually save time and money in the end.
Studies which have been inadequately piloted or not piloted at all, will find that
a great deal of effort has been wasted on unintelligible questions producing
unquantifiable responses and uninterpretable results. Moreover, dozens of
administrative matters concerned with sampling and fieldwork that 'could not
possibly go wrong', will go wrong.
Pilot work can also be immensely rewarding There is an intellectual
- --- - -
.
.
challenge
.
Types of interviews
in conceptualizing and re-conceptualizing the key aims of the study and in
making preparations for the fieldwork and analysis so that not too much will go probably no other skill is as important to the survey research worker as the
wrong and nothing will have been left out. There is satisfaction in seeing the ability to conduct good interviews. No amount of sophisticated scale-building or
emergence of a short, reliable scale from the mass of pilot data. And there is statistical analysis can rescue a research project in which the conceptualization
gratification in finally producing a really effective set of multiple choices which and instrumentation have been built on poorly conducted exp-loratoryinter-
resuondents can use and which neatly embody the purpose - -.
of the question. Pilot views; this is one phase of a research project which it would be unwise to leave
work can produce some nasty surprises, but it is never dull. to someone else. The interview, unlike most other techniques, requires
interpersonal skills of a high order (putting the respondent at ease, asking
questions in an interested manner, noting down the responses without
Selected readings upsetting the conversational flow, giving support without introducing bias); at
the same time the interviewer is either limited or helped by his or her own sex,
198 1 Census Post Enumeration Suroey Report, HMSO, 1985. apparent age and background, skin colour, accent etc. When taken seriously,
interviewing is a task of daunting complexity.
First, let us make a few distinctions. Interviews may be used for many
different purposes - we have press interviews, therapeutic interviews,
employment selection interviews, interviews making requests or stating
demands and so on. But the interviews with which we shall be concerned are
different from all these. They are essentially of two kinds:
As is evident from their titles, and from the list of research stages we considered
in Chapter I, these two types of interviews generally enter the research process
at different points: first, the exploratory interviews and later (if at all) the
standardized interviews. They also have quite different purposes and have to be
carried out in quite different ways.
An interview is not an ordinary conversation, although the exploratory
interview appears similar in some respects. In some societies the concept of a
social research interview either does not exist or is vigorously resisted precisely
because it is not an ordinary conversation. Until it is accepted as a socially
approved phenomenon, with its attendant roles and expectations, it may well
outrage potential respondents for many reasons: they resent the intrusion by a
66 Questionnaire Design, Interviewing and Attitude Measurement I The exploratory interview 67
complete stranger; they do not want to be so accurate or attentive in their We now come to the actual interview, with the interviewer and the interview
responses; they want to share in the control of the interview and ask questions schedule in interaction with the respondent. The crucial assumption here is that
of the interviewer; and they fear the potential use to which their responses 1 of 'equivalence of stimulus: that is, we have to rely on the interviewer's skill to
might be put. Or, conversely, they may feel enormously flattered at being ,
I
approach as nearly as possible the notion that every respondent has been asked
interviewed at all; engage in elaborate displays of friendship and hospitality; the same questions, with the same meaning, in the same words, same
involve the interviewer in personal or family matters; and seek to strike up a I intonation, same sequence, in the same setting and so on. That is why we call
lasting relationship. Every interviewer must realize that these problems exist these interviews 'standardized', though to achieve this kind of psychological
and that people have a perfect right to feel this way; indeed, the ethics of
conducting interviews are a continual topic of lively controversy. Every
1 'sameness' is a manifest impossibility. What is needed, rather, are ways of
making the question mean the same for each respondent even if this should
interviewer must also learn how to cope with the biases and distortions which
these problems create - always assuming that some kind of interview
/ require somewhat greater flexibility in the role of the interviewer. Finding this
balance between flexibility in reacting to the respondents' answers and
eventually takes place! For many people, being interviewed requires the learning
of a new social role (the 'good respondent' role) in a new social exchange setting // standardization in the question asked is the particular skill of the good
interviewer.
The purpose of the exploratory interview is essentially heuristic: to develop
and the interviewer, too, has to learn and adopt a social role which may be quite
unlike his or her normal persona.
If an interview is not a conversation, then what is it? The purpose of all research
1 ideas and research hypotheses rather than to gather facts and statistics. It is
concerned with trying to understand how ordinary people think and feel about
interviews is to obtain information of certain kinds. This information may be in
the form of factual replies to factual questions, or responses to attitude scaleitems,
/ the topics of concern to the research.
Often these are emotionally 'loaded' topics and only a highly-skilled depth
or ideas and feelings, or percepts and expectations, attitudes and the like. The interviewer can get people, over the period of an hour or longer, to talk freely
respondents may, or may not, 'have' t h s information, or they may 'have' it but and with some degree of insight about their thoughts, feelings and formative
be unable, or unwilling to communicate it. Although words and sentences are experiences. Here nothing is standard. At most the interviewers will have a
exchanged in both directions, an interview is essentially a one-way process. 'hidden agenda', a handful of headings or topics around which they will seek to
Indeed, if it should become a two-way process of communication (more like a direct the interview as unobtrusively as possible. In the main fieldwork stage,
genuine conversation), it will lose much of its value because of the biases many hundreds of respondents will typically be interviewed in a standardized
introduced by the interviewer. Respondents get very little out of a standardized way; by contrast, at the earlier exploratory stage, perhaps no more than thirty
interview - perhaps a gift, a slight degree of interest, a little ego-boosting and to forty free-style interviews will take place. It would be very foolish even to try
a general feeling that they have been of help. Interviewers in a standardized to quantify so small a number and, besides, these respondents are unlikely to be
interview also get very little out of it; the data will go back to research properly representative of the survey population; nor will they all have received
headquarters and, at the most, they will see areport on the study, perhaps months the same questions, let alone been asked the same questions in the same way.
later. There is (apart from money) only the intrinsic interest in meeting many The job of the depth interviewer is thus not that of data collection but ideas
different people and in ajob well done. The depthinterviewer probably gets rather collection. The primary objective is to maintain spontaneity; the ideal free-style
more out of it since he or she will be more deeply involved in designing the interview would consist of a continuous monologue by the respondent on the
topic of the research, punctuated now and again by an 'uhuh, uhuh' from the
both kinds of interviewer must 'switch off' their own personality and attit desI
research and developing hypotheses andinstruments. But in order to do their job,
interviewer! The interviewers will seek to reduce their own role to an absolute
minimum, to avoid leading the respondent. If something is not clear a 'non-
(this can be very exhausting) and try to be unaffected by circumstances, by their
attitude to the topic or the respondent, or by personal involvement. directive' prompt will be used such as 'yes, yes' or 'I see, yes, do go on', or
summarizing what the respondent has just said, or maintaining a pleasantly
expectant silence. Depth interviewers must, as the saying goes, 'listen with the
Standardized interviews versus depth interviews third ear'. They must note not only what is being said but also what is being
The purpose of the standardized interview in the typical large-scale survey is omitted; must pick up gaps and hesitations and explore what lies behind them;
essentially that of data collection.We are at t h s point in what might be called the and must create an atmosphere which is sufficiently uncritical for the
'mass production' stage, as distinct from the 'research and development' stage. respondent to come out with seemingly irrational ideas, hatreds or
The various research objectives and hypotheses have long since been formu- misconceptions.
lated; the interview schedule and the wording of the questions have been It is essential for the exploratory interviews to be recorded on tape. In this way
exhaustively tried out in pilot work; the sample has been drawn; the field force they can be analysed in detail afterwards, for there is much that will have
recruited (possibly); and arrangements made for coping with refusals and escaped the busy interviewer in the stress of the actual interview; also the tapes
otherwise unobtainable respondents, with fieldwork checks and the return of can be examined by more than one person. A useful set of exploratory
the completed schedules for data processing. interviews can greatly broaden and deepen the original plan of the research,
68 Questionnaire Design, Interviewing and Attitude Measurement The exploratory interview 69
throw up new dimensions to be studied, suggest many new ideas and spontaneous reactions that are wanted, not carefully thought out positions.
hypotheses, important differences between groups of respondents and so on. As If the respondent is willing and can spare the time, details can then be settled
an added benefit the interview tapes will produce a rich store of attitudinal and about time and place, possible child-minding and other arrangements, special
perceptual expressions, on which questions and attitude items can be based. means of transport and so forth. Recruitment interviewers, like all others, must
Thus, depth interviews help in the formulation of the research problem, in the carry identifying letters and offer respondents some printed information about
articulation of dimensions and hypotheses and in the details of instrument their organization and an address and phone number in case they have further
building. Only in the eventual full-scale study will it be (possible to test the questions.
validity of the hypotheses thrown up in the exploratory interviews. ~ e c r u i t i npeople
~ for depth interviews often meets with difficulties such as
shift work, busy people being unable to spare the time, reservations about being
lprobed' by a psychologist and so on, which may all lead to bias in the sample. It
The depth interview would be unrealistic to expect a depth-interview sample to be a precise
Let us now enter a little further into the problems of the depth interview. microcosm of the intended sample in the main survey. What we need is a sample
First, whom should we interview? In principle, we should wish to interview a whose outlook, attitudes, percepts and experiences are likely to cover a similar
judgement sample, a small sample that is typical of the respondents whom we range and variety in order to avoid unanticipated responses later, when
aim to question in the main survey. Exact representativeness is not usually questioning the main sample.
necessary, but we need a good spread of respondent characteristics so that we Where should depth interviews take place? It is possible to conduct depth
can reasonably hope to have tapped probable respondents of every kind and interviews in the interviewer's home, in the respondent's home, or at the
background. Sometimes we may aim to set up two contrasting sub-samples; for respondent's place of work, bu it is unlikely that a long enough period without
example, when interviewing widows about their husbands' death and funeral or interruptions and distractions %ill be available. Home interviews are unavoida-
cremation (a subject that obviously calls for great skill and delicacy), we might ble with some groups of respondents, but it is generally preferable to conduct
depth interviews in a meeting room in a local hotel (to reduce travelling
wish to make a distinction between widows whose husbands died suddenly or
unexpectedly and those whose husbands died after a long and serious illness, and distance), in an office hired for the purpose in a nearby district, or at rooms of the
research company or academic department. It is best to avoid settings that may
so we may divide our intended sample of depth interviewees accordingly. In
be perceived as unpleasant or threatening by respondents; for example, a school
consumer research, we might wish to compare users and non-users of a
head's study, a hospital, a room above a pub or a betting shop. Potentially noisy
particular product or service. Sometimes, too, we might wish to conduct depth
interviews
-----
- manager, and the personnel
with 'key informants' such as the- works settings should also be avoided in case this causes interference with the tape
officer in a factory, or funeral directors in the above example. recordings.
How many depth interviews should we conduct? There can be no definitive There should be an ante-room, where a receptionist can regulate the flow of
answer to this question but thirty or forty interviews is probably typical. In the interviewees, deal with children and with the payment of fees and expenses,
phone latecomers, arrange for transport and so on. Refreshments should be
process of conducting a series of depth interviews, in which usually only two or
three especially experienced and trained interviewers are involved, it generally available.
becomes obvious when the series has reached the point where no new ideas are The setting for the actual interview should be private, quiet and comfortable
and not intimidating. Some thought should be given to the positioning of easy
emerging; a quick consultation will then bring the series to a close. Obviously
chairs, coffee table, recording equipment, lights and so forth and to the question
depth interviews are costly and time-consuming, and so there are usually
of smoking or non-smoking. If there is a desk in the room it should not form a
extraneous pressures to reduce numbers to a minimum. But quality, rather than
barrier between the interviewer and the respondent. Everything should be done
quantity, should be the essential determinant of numbers.
to create a comfortable, unhurried and relaxed setting for a private, confidential
How should the respondents be obtained? Ideally each respondent should be
talk; anything that might upset or disturb respondents or make them feel
recruited by a field interviewer. One or more field interviewers should be sent
pressed or intimidated should be avoided. The respondents should come away
out with something like a quota sample to fill (for example, five middle-class
with a vague feeling of pleasure at having been of help and having had an
women over sixty who have lost their husbands in an accident or in some other
interesting conversation with an attentive professional, though preferably
sudden way within the past six months). They should explain to potential
without a clear idea as to the exact purpose of the interview; they should never
respondents that the interview will last about an hour or perhaps longer; that
go away angry or upset, or feel that they have been the subject of a painful
the interviewer will be an experienced professional who will maintain confiden-
inquisition.
tiality and high ethical standards; that the questions will not be of the market-
By whom should depth interviews be conducted? Formal training or academic
research type but will range widely over many topics; and that their expenses
qualifications in one of the social sciences are no guarantees of competence in
will be paid, together with a modest fee (if any) for their assistance. If possible,
this field. The ability to present a somewhat neutral presence, a friendly and
the respondents should be given only a vague idea of the central topics of the
interested manner, extensive experience and insightful 'traffic management' are
interview (for example 'it's about the ways people spend their leisure'); it is
70 Questionnaire Design, Intemiewing and Attitude Measurement The exploratory intemiew 71
and to present a measure of authority and an assurance of confidentiality.Tone wider implications of the research for such topics as gender roles, social change,
of voice, a pleasant and polite manner, deportment, choice of dress, the media impact and so forth.
management of personal space, an acceptant and non-judgemental attitude and What does the depth interviewer actually do? The depth interviewer is often
a willingness to listen should be among the interpersonal skills; these will frantically busy, without being allowed to show this. Having welcomed and
encourage respondents to talk freely on personal matters, while not giving the thanked
. the- respondent after being introduced, the respondent must now be
impression that they can get away with trivial insincerities or waste the seated and put at ease, perhaps with the help of some refreshment and the offer
interviewer's time. High ethical standards should prevail; no respondent should of a cigarette (or not, depending on smoking policy). The interviewer will next
be afraid to produce self-incriminating or embarrassing feelings or information, produce an agreed and rather vague statement of purpose such as: 'well, as you
and if a respondent gets angry or tearful the interview should be gently may have been told already, today we are conducting some discussions about
discontinued and the respondent helped to regain composure. opinions on the media (health foodslthe trades unionslthe causes of
What is the 'hidden agenda'? As we have seen, the purpose of the depth unemployment). We are planning a major survey on this topic, and we have
interview is to collect percepts and ideas and to improve the conceptualizationof asked you along today so that we can benefit from your views and experiences
the research problem. Interviewers must be well briefed so that they fully in planning our research. The interviews are entirely private and confidential,
understand the objectives of the research. Since there are no fixed questions, your name will not be linked to anything you say here, and I have not been told
each depth interviewer will be equipped with a list of general topics or areas anything about who you are or where you come from. Most people find these
around whch the interview should be conducted; these topics will be derived interviews very interesting.'
from the first attempts at conceptualization and will be added to from time to Before going any further, the interviewer must now, if possible, secure the
time during the course of the interviews. For instance, if we may return to the respondent's permission to record the interview on tape. Usually, there will be
earlier example of a study of recent widows, it is obvious that we shall need to a microphone in plain view (turned towards the respondent, not the inter-
ask a number of factual questions to do with the cause of death, cremation or viewer) and the recorder will be under the interviewer's control. In industrial-
burial, religious affiliation and so on. We shall need to explore ideas about death ized countries most respondents will be familiar with sound recorders and
and 'after-death', memories of the deceased, responses of the extended family, readily give their consent when asked, 'do you mind if I tape our conversation?
other sources of social support, and (more delicately)causes of anger, hostility or It helps me to remember afterwards what you said, and' (disarmingly)'it saves
disappointment, loneliness and fears, perhaps a sense of relief, financial matters me taking notes.' In less-developed countries more explanations may be needed
and the costs of burial or cremation and many others. These might constitute and various suspicions may have to be allayed. The recorder must not be turned
the initial hidden agenda. During some of the early interviews we may also find on until permission has been obtained. If the respondent refuses to consent to
that the counselling sometimes offered by the funeral firm seems to be a tape recording, the interview may proceed with the aid of some ultra-rapid
important to some, that there are interesting attitudes to the formalities note-taking, or the interviewer may decide to terminate the interview as soon as
required after death, such as obtaining the death certificate, and that matters of politely possible. (If this happens repeatedly, this may cause a bias.) Most
appropriate funeral dress, floral tributes and the disposal of the deceased's respondents soon forget all about the microphone, though some may ask to hear
clothes are all 'productive' areas to probe in an effort to conceptualize more fully a portion of the tape played back to them.
what the death of a husband and the sudden acquisition of the new status of The interviewer has already been busy up to this point, gaining 'first
widow 'mean', in a given society. Such topics will now be added to the hidden impressions', planning how to start the interview proper, perhaps pouring
agenda. coffee or coping with a recalcitrant tape recorder while going through the above
The hidden agenda is only 'hidden' in the sense that it should not be too introductory phases. To gain a few moments of respite, and to get the
obvious to the respondent. The interview should move naturally from topic to respondent settled into a 'question-answering mode', the interviewer may open
topic, maintaining the fiction of an interesting conversation. The interviewer the next phase with a wide-open, highly projective general question ('projective'
may start with any of the topics on the agenda and proceed in any order. It is in the sense that the question means whatever the respondent chooses to make
often not possible to cover all the topics in a single interview; it is better, it mean, which in turn may be interesting or significant) such as: 'Now, first of
therefore, to obtain some rich material on a handful of topics, rather than to all, would you like to tell me a little about yourself?' Such an opening question
press on at a relatively superficial level in order to cover every point. should sound chatty rather than intrusive. In general the first few questions
As conceptualization and re-conceptualization proceed, it is most helpful for should be spoken quite slowly, to help the respondent to get used to the
interviewers to exchange ideas and tape recordings; for research students or interviewer's voice and manner.
other aspiring depth interviewers to have frequent meetings with their Such a broad opening question may produce a small flood of information at
supervisors; for all concerned to exchange short notes, write-ups or quotations several levels. Up front will be something, perhaps, on occupation, marital
from interviews. This should help to expand and sharpen ideas about which status, children (if any) or dwelling area. Immediately behind this may be noted
72 Questionnaire Design, Interviewing and Attitude Measurement The exploratory interview 73
the 'presentation of self' aspects. (Is the respondent nervous? Trying to impress banks', the interviewer may now, at last, raise a quizzical eyebrow and say: 'The
us with status or possessions? Resistant or secretive? Has something obviously banks? Well, what about the banks? Tell me more.'
been left out; for example, ethnicity, physical handicap?) Deeper still are other Besides this kind of 'traffic management', the interviewer is also following
domains such as how (at this point) the task of being interviewed is perceived by parts of the hidden agenda in what will hopefully seem like a natural order. Here
the respondents; perhaps what they think the interviewer wants to hear or again, judgement has to be used when a promising lead opens up: to follow the
know about; how they construe their own identities, at least for now; and what lead or to stick to the agenda? In an interview on self-medication (buying non-
systems and language of social categorization they use, for example: 'We live in prescription medicines at a chemist's) a young woman was asked what she
a council flat across the river.' would do in the case of a sore throat. She started to answer, then stopped herself
It is worth mentioning that this opening question, 'Now, first of all, would you and said, a little pompously: 'No, I would do nothing, for the throat must take care of
like to tell me a little about yourself?' can be answered, even by a co-operative itself'. In the rush of the moment some interviewers would have missed the cue,
respondent, with the monosyllabic answer, 'Yes'. This gets us no further; the but it turned out that this young woman perceived her body as largely divided
question violated one of the elementary rules of depth interviewing, namely to into two zones: organs and areas of which she had to take care, such as the hair,
avoid asking questions that can be answered by simply saying 'Yes' or 'No'. the skin, the nails; and parts that had to 'take care of themselves', such as the
Moreover, instead of gaining the busy interviewer a few moments' respite and kidneys, the lungs, the intestines. Then there was a third, borderline zone of
perhaps some good jumping-off points for further questions, it suddenly organs which might, or might not, take care of themselves; this zone contained
requires the formulation of further questions. Or worse, the respondent may most of the body's orifices, including the throat. Thus, her buying behaviour at
smile helpfully and say, 'Yes - what would you like to know?' So now the the chemist's was determined by her perceived obligation to care, or not to care,
interviewer has fallen into the trap of role reversal and will have to 'project' some for different organs of her body. The throat seemed to be in a borderline
of the interests of the research while trying not to get flustered! O r else deflect category, and when she felt that 'the throat must take care of itself', she would
the question back onto the respondent by playing a further projective card: 'Well, not buy any remedies for it. It took some time to tease these ideas out of the
tell me anything about yourself that you think I might need to know.' Oh dear! respondent, who was well aware that these were unorthodox notions, but this
Our interview has not got off to a good start. diversion from .the hidden agenda proved worthwhile when, in many subse-
Assuming, now, that the respondent has given us a few personal details, we quent interviews, other respondents also came up with variations on the same
next have to engage in a spot of rapid mental traffic management. We cannot set of ideas.
follow up every piece of information at once, so we have to make a choice. This example also illustrates the third main below-the-surface activity of the
However, we also have to remember most of the other items, to follow up later; interviewer, besides traffic management and following the hidden agenda:
perhaps we should jot these down. This is a situation that will recur throughout maintaining rapport at the right level so that the respondent will keep tallung. In
the interview; the respondent touches on more than one topic (or more than one the above example, if the interviewer had manifested surprise, ridicule or had
idea occurs to us, while listening), so we have to choose what looks most likely merely raised an eyebrow, the respondent might have closed up. Only by
to be a productive topic, realizing all the time that we might have done better to maintaining a receptive, permissive and non-judgemental attitude throughout
choose a different topic from the respondent's reply, one that we are now hoping can the interviewer hope to obtain insights into motives, ideas and percepts
to remember and to slip into the conversation naturally at some future moment. which respondents know to be 'irrational' or self-incriminating but which
A co-operative and voluble respondent can leave the interviewer with a wide nevertheless guide their behaviour.
mixture of leads to follow up. ~nterviewersmust not only try to get the respondents to 'flow' freely by
This type of 'traffic management' goes on throughout the interview. It need encouraging them and by rewarding them with comprehension; sometimes the
not always remain hidden below the surface. Sometimes we can say: 'Now you flow has to be stopped, or diverted, when the respondent is spending too much
will remember telling me a little earlier that you disliked XYZ. Can we now come time on issues which are far removed from the purposes of the research. This is
back to that? I wonder what makes you feel this way?'and thus deliberatelyswitch not easy to do without upsetting the respondent, but an attempt could be made
topics. The reverse may also happen. A male respondent may, for example, talk by saying: 'Shall we come back to t h s later, after I have finished my questions?'
about his business and add an aside such as: 'But of course you can never trust At other times the respondent may try to turn the interview into a conversation
the banks.' Clearly he is throwing the interviewer a 'lead' and hopes that it will by asking the interviewer questions; these could likewise be deflected by
be picked up. The interviewer may wonder, and then decide to test the strength suggesting, 'shall we come back to that later?' while noting and responding to
off eelingbehind the throw-away remark by not asking a question about the banks. any of the respondent's immediate anxieties such as 'who else is going to listen
Sure enough, several exchanges later the respondent may say again: 'But of to these tapes?'
course, we all know that the banks can't be trusted.' The interviewer may, quite How are questions formulated during a depth interview? Part of the answer
deliberately, decide once again not to rise to the bait until the same sequence is is that, in a depth interview that is going well, there should hardly be any need
repeated once more. Having ascertained the strength of feeling in the respondent for questions as such. The interviewer may merely suggest a topic with a word,
on this topic by observing his attempts to divert the interview to the topic of 'the or an unfinished sentence left trailing, and the respondent will 'take it away' and
74 Questionnaire Design, Interviewing and Attitude Measuremen! The exploratory interview 75
will a rounded and personalized response, punctuated by sounds of indeed, silence is often a very effective yet non-directive press to go on. Or,
encouragement from the interviewer. A depth interviewer has to engage in a taking more of a risk, we might repeat back to the respondent what has just been
little bit of acting from time to time rather than try to ask 'correct' questions. said. This may seem odd, at first: will the respondent not get angry if we just
Thus, there is probably no need to ask: 'Now what about your children? Do you repeat 'mmm, you say women should not go into politics?' However, it seems
have boys, or girls or some of each? How old are they? And how do you feel that, if done naturally, respondents experience this as evidence that they are
about them? Do you have a favourite?' The interviewer could merely look up getting across and as a signal to develop their views further. But there is always
and say, quietly, 'Children. . . ?' and await a reply as construed by the the risk that in repeating a view we distort it: for example by saying 'I see; you
respondent. This also avoids the danger of leading, of putting ideas into the say women should be kept out of public life' or worse: 'I see; you say women
respondent's mind or pre-determining the lines of response. should stay at home and mind the kids' - which is not at all what was said! The
When questions have to be asked, they should be as open and as projective as incomplete sentence can also be used as a non-directive probe, for example 'I see;
possible. We should try to avoid formal and leading questions such as: 'What do so you feel that women and . . . politics . . .' (on a rising note), leaving the
you consider to be the appropriate role for the Church when someone has died?' respondent to complete the sentence and carry on. A less oblique probe which
or 'do you keep a bottle of disinfectant in the cupboard under the sink?' but is still non-directive is the use of a gentle inquiry such as 'what makes you say
explore each area more obliquely to allow the respondent maximum sponta- that?', 'could you explain further?' or 'please tell me more.'
neity. We might start by asking, say: 'Now what about funerals? Let's discuss In some kinds of depth interviews real objects (for example consumer
how you feel about them.' This formulation is designed to let the respondent, products; toys; car badges), or pictures of faces, scenes or objects can be
not the interviewer, determine which aspects of a funeral arouse strong feelings; introduced for discussion and for sorting into categories (for example, alcoholic
also to see if the role of the Church is mentioned without prompting. In the case versus non-alcoholic drinks; modern versus traditional). Often the same topic is
of the disinfectant question, we might start by getting the respondent to say approached more than once during an interview, in different ways or from
what is meant by the term'disinfectant' and to mention some brand names; then different angles.
discuss family responsibilities and see if a specific member of the family is held Very occasionally, after a topic has been explored in a free-style manner, a
'responsible' for the supply and use of disinfectants, and where they are kept and deliberately directive probe will be used; for example, 'some people say that more
why. Some men, for example, will not use bath salts because they think they are goods should be carried by road. How do you feel about this?' When driven to
'sissy' and will use quantities of disinfectant instead. Thus, the use and location use such directive questions, it might be better to offer two or more points of
of disinfectants in some homes is closely linked to symbols of manliness. view, such as 'some politicians are campaigning to have more goods carried by
In depth interviewing, every effort is made to get respondents to express their road, while others want more goods carried by rail. What do you think?' But the
own ideas spontaneously in their own words. It is only too easy for an question remains leading, in that it assumes the respondent must have a point
inexperienced interviewer to ask questions which are subtly leading (or of view and that it can only be one of the two choices offered in the question.
sometimes not so subtly!) and so make respondents aware of what may be If the interview has gone fairly 'deep', perhaps bringing up old resentments,
expected of them. Thus, the question 'do you think that a clergyman has no role painful memories or hurtful experiences, then the interviewer should offer
at a modem funeral?' is highly leading (note especially the word 'modern' and its acknowledgement: 'I appreciate that talking about things like this is not easy' or
implications); but so is 'how do you feel about abortion on demand?' or 'do you 'knowing something about your personal experience of X will be very valuable
think our country should go to war to save its oil supplies?'The best way to learn in guiding our research.' It might also be advisable to take a few minutes to
to avoid leading questions is to listen to your own tapes - this can be quite soothe the respondent by discussing more superficial topics, in order to ease the
mortifying but is a much healthier corrective than criticism from colleagues. Yet stress of the situation and make the respondent feel more comfortable.
under the stress and immediacy of a difficult depth interview even the most How should a depth interview be concluded? Eventually, the interviewer
experienced interviewers will occasionally lapse into leading questions. might indicate that the interview is over, for example by saying, 'well, I think we
Depth interviewers therefore need to practise the non-directive approach. They have covered everything I needed to ask you. Thank you very much, you have
must exercise great patience, they must try to interfere as little as possible and been most helpful,' perhaps followed by 'is there anything you would like to ask
let the interview 'run itself. Above all, they must cultivate the art of the non- me?' or 'do you have any questions?' After answering these, the interviewer
directive probe. Suppose the respondent has just come to a momentary halt, might get up, formally switch off the tape recorder, thank the subject once more
saying: 'And so I don't believe that women should go into politics.' We should for attending and move towards the door. Sometimes, at this very last moment,
avoid the temptation to disagree with the respondent or to enter into an the respondent will make a revealing comment or offer a confidence, perhaps
argument. Our job is to get the respondents to reveal their inner thought out of relief that the interview is over. The interviewer should be mentally
processes -but how are we to do this? Here is where we need the non-directive prepared for this and not hustle the respondent to the door, for often these less-
probe. For example, we might merely offer a tentative 'uhum, uhum', implying guarded 'off the record' remarks can be especially valuable. They should, of
that the respondent should continue to develop this line of thought. Or we course, be noted down as soon as the respondent has left.
might just say nothing, and merely smile expectantly. Most respondents find How should the interviews be processed? The interviewer should take a few
silence in an interview unbearable and feel that they have to continue talking - minutes after each interview to jot down ideas and suggestions that have been
76 Questionnaire Design, Interviewing and Attitude Measurement The exploratory intemiew 77
'Where's your drink, Doreen?'I said, 'I think I've left it over there. I've had enough. I must
stimulated by the interview as well as anything that might have affected the
interview, for example any intervention by a third party in a home interview. have had one or two before.' Now, I'm lying, ain't I? Two! I'd had bleedin' none, of a
sudden, a waiter came up to me. 'Oh, !here you are,' he said, 'I've been looking for you.
As soon as possible, the interviewer and any available colleagues should listen Here's your bitter lemon.' So, right, I takes the bitter lemon. 'What's the matter?' says
t o each tape, for different people note different underlying themes in an interview Emily. I said, 'I don't drink. I don? want to make myself look like a fool. If I start drinking
tape. Usually this takes considerably longer than the duration of the interview 11' do the strip tease in here.' So, out we goes into dinner.
because the tape will be frequently stopped and sometimes re-wound a small Now, each table has got so many staff and so many partners, two partners to a table.
distance to go back over a passage. First, attention has to be paid to operational Who should be on my bloody table but two senior partners. I thought, Jesus bloody
matters; for example, should new topics be included in the hidden agenda(s)? Christ! So, as we sit down, this one started into me. I hadn't been at the firm long, so he
Should more vegetarians be interviewed, or some other group? Are new ideas didn't know a lot about me. 'Are you settling in here?'he said. 'Yes, I'm enjoying itl1Isaid,
'and I think this is lovely.'I thought, if I could get out of here I would, but I couldn't, could
no longer emerging, in which case should the depth interviews be discontinued? I? So, I'm sitting there. This is my first big 'do' and look at the 'do' I'm at, and itS all bitter
Next, a short resum6 of each interview should be prepared, highlighting some lemon.
of the key topics discussed. After that, an attempt should be made at You have different drinks with every bit of the meal, you know, and I thought, Jesus
interpretation, both at the personal and at the theoretical level. For example, Christ, how much more of this are they fetching round! Each time I'm having bitter
silences and omissions might suggest painful or very personal issues. Insight and lemon. This bloke, at the side of me, out of the blue he said to me - words to the effect
intuition are important at this stage, but the 'correctness' of any particular that do I not drink, or do I not drink for a reason. Now, that bloke could have been
interpretation is irrelevant. Remember that the prime purpose of these genuine, but straight away I think 'I'm an alcky, ain't I?' I had the shakes from top to
interviews is heuristic: to generate further ideas about topics and variables that bottom. I said, 'Oh no, I've never drunk. It doesn't appeal to me. Other people like it, but
should be included in the research plan, links that should be explored, attitudes I couldn't care less, as you can see.' 'Well,' he said, 'it surprised me to see that you don't
and motives that might determine some behaviours. take a little wine with your meal."Well, it's not being used to it,' I said, ' I'm not in your
class. I'm married to a greengrocer, not a lawyer.' I thought, right, I'm what I am and you
What further use may be made of the depth interviews? The tapes can be an can't alter me. I'm a Battersea girl. He thought this was funny, but I was shaking.
invaluable source of useful question wordings and attitude statements. Now, after we moved from our tables, this bloke that was sitting near me, he went up
Research workers tend to become too intellectual, and to develop the language to two senior partners. Now, he could have been talking business, but now and again they
of specialists; it is generally much better if attitude questions and statements (see looked round at me. Knowing what heil said to me, I thought, maybe they're talking
Chapter 10)can be put in the respondents' own, every-day language. Therefore about me. Now, when I go in Monday, I ain't got a job!
the tapes should be combed for expressions and sayings that could be used as At 12 o'clock, when the car came for me, I got in and I broke down and cried. When I
attitude items in a questionnaire - o r in a subsequent depth interview. Depth got home, I woke up me husband, and next morning I rang up my doctor, and I was still
interviews should not be regarded as arcane or esoteric, bordering on the shaking and crying. My first words to him were 'I went to a "do" last night, but I haven't
psychoanalytic and of use only in research that is highly 'academic'. They can be been drinking!'
very useful in such conventional and practical areas as market research, opinion This is, admittedly, an unusually vivid and fluent account, told without any
polling, product development, patient follow-ups, architectural research or prompting. The incident had taken place some five years earlier, yet it was fresh
transport studies. in the respondent's mind, perhaps because her adjustment was still somewhat
Depth interviews can often lead directly to a useful conceptualization and the fragile. The purpose of this series of depth interviews was to develop further
building of pilot instruments. Here is an extract from a depth interview with a ideas about this kind of fragility and the dangers of relapse. How do ex-alcoholic
south London woman in her fifties who had been an alcoholic and was now patients manage to stay off drink? What are their subjective experiences, their
taking part in a follow-up study: fears and perceived risks, their coping styles, their social supports? This series of
initial interviews, together with ideas from the relevant literature, eventually led
1ain't gonna say I haven't been frightened. I'll tell you one experience I had. to a conceptual framework and the decision to measure a number of variables by
I'd started work in my place in January, and in June every year we have a Savoy 'do1. means of structured inventories. O n e of these was an inventory of 'dangerous
Now, that's laid on just for the staff - no husbands, no boyfriends, only the staff. Well, situations' (dangerous to sobriety, that is). The actual items were obtained from
I went, and that was my first social gathering off the bottle. I'd only come out of hospital interviews such as the above and from pilot work using a sentence completion
at Christmas. Well, I goes in there and, as you know - well, I don't know if you do, but
itls all la-di-da, you know, top-hat-and-tails sort of thing. Well, I went in style. I had a technique (see Chapter 4). After editing and further pilot work, two dozen items
Mercedes to take me and I had a Mercedes to fetch me. Well, that's my husband. 'You're were randomly arranged in a structured inventory of which the following is an
going in style. You're going to be the same as everybody!' extract:
Now, when I get in there, all the bosses are there, all the VIPs from the firm, and who Here are situations which some people have experienced as being dangerous to their
should walk up to me but one of the senior partners. 'Oh, glad to see you, Doreen.' I was staying off drink. Which of these may have been dangerous for you during the pasf four
shaking from head to toe. 'Got here all right? Now, what would you like to drink?'I said, monfhs? There are four boxes. 'Very dangerous, Quite dangerous, A little dangerous, Not
'I don?drink, but I'll have a bitter lemon.'I think I was the only non-drinker there; So I said, at all'. Please tick that box which comesclosest toyour feelings about those situationswhich
'Excuse me, Sir, therels Patricia over there,' and off I goes. I goes up to them and it's
78 Questionnaire Design. Interoiewing and Attitude Measurement The exploratory interview 79
may have been dangerous to your staying off drink during fhe past four months. There are This inventory was twice subjected to factor analysis (see Chapter 9) and
no right or wrong answers or trick questions. We want to know how you feel. PLEASE yielded three independent factors:
BE SURE TO ANSWER EVERY QUESTION. Please put only one tick for each question.
Factor I - infernal situations (moods and feelings), for example when I feel tensel
'SITUATIONS DANGEROUS TO MY STAYING OFF DRINK' depressedlvery angryltired;
Factor II - external situations (social occasions. holidays). for example when Iin on
VERY QUITE ALITTLE NOT AT
ALL
holidaylon special occasionslat Christmas;
DANGEROUS DANGEROUS DANGEROUS
Factor 11.7 - lessened cognitive vigilance, for example when I start thinking that just
one drink would do no harm.
12. When I pass a pub or off-
licence
In this way, when this inventory was administered in the main study to over 300
13. When I'm with other people
ex-patients, each respondent could be given three scores which, in turn, could be
who are drinking related to other variables.
Thus the depth interviews supply the ideas and the conceptu&zation of the
14. When I feel no one really problem, they help to identify the variables to be measured and they provide
cares what happens to me many of the items for new scales. The process helps to ensure that, despite the
requirements of reliable measurement, we retain the subjective framework of
1 15. When I feel tense I I I I I our respondents.
16. When I have to meet people
Selected readings
Banaka, W.H., 1971, Training in Depth Interviewing, Harper & Row, New York.
ONGROUP INTERVIEWS
Krueger, Richard A., 1988, Focus Groups, Sage, London.
Interviews versus questionnaires: advantages of interviews
Let us now return to a consideration of standardized interviews. We may well
ask: why use interviews at all? To this there are a number of answers, but let us
remember the overriding condition of relevance or appropriateness: one cannot
say that interviews are always good or always bad, but rather that interviews are
preferable for some problems, or under some conditions, and not others - and
often, it has to be admitted, the choice will eventually be made for quite
extraneous reasons such as costs or pressure of time.
Interviewers come into their own when we need to ask numerous open-
ended questions, or open-ended probes, and where the interviewer has to record
verbatim the answers given by the respondents. Such open-ended questions are
important in allowing the respondents to say what they think and to do so with
greater richness and spontaneity. Obviously, interviewers have to be trained
and instructed most carefully in the flexible use of probes, and the probes
themselves may have to be standardized.
It is often suggested that another advantage of using interviewers is that of
improoed response rates. A postal questionnaire may easily produce a response rate
below 40 per cent; interviewers can generally do much better than this, but not
always. We have to distinguish between quota samples (trying to find, say,
twelve middle-class men over the age of forty-five) and probability sampling,
where the interviewer is given a list of names and addresses (for instance, from
the electoral register). In the latter case a great deal of time and effort may have
to be expended in repeated call-backs before a new name is substituted.
Likewise, quota sample interviewers may readily meet a very high refusal rate
as well as other difficulties in filling their quotas. However, in the final analysis
what matters is not only the total number of completed questionnaires but,
more importantly, the question of bias in the sample of respondents. In that
respect, interview surveys based on probability samples offer a clear advantage
over postal questionnaires. .
A third set of advantages in the use of interviewers is that they can give a
prepared explanation of the purpose of the study more convincingly than a
covering letter can; will more easily reach less well-educated respondents; help
the ones with reading difficulties; offer standardized explanations to certain
82 Questionnaire Design, Intemiewing and Attitude Measurement Standardized intemiems 83
The cost factor also enters the data-processing stage: since interviewers are
problems that arise; prevent many misunderstandings; and maintain control
over the order or sequence in which the questions are answered. These are some used particularly where many open-ended questions have to be asked, there ~ 1 1
be a major and costly coding operation allied to any study that uses interviewers.
very down-to-earth, practical advantages. Generally, it can be said that the longer,
Coding, which we shall be considering in Chapter 14, is the name given to the
the more difficult and the more open-ended the question schedule is, the more we should prefer to
process of developing and using classifications for the answers to each question.
use interviewers.
The costs have to be assessed in money terms but also in time: unless a large,
Many research workers also like to use interviewers because they feel that
permanent interviewer force is engaged, most studies using interviewers will
interviewers are more 'valid' in some sense. 'After all,' they would argue, 'the
take weeks if not months to complete -and the coding may take longer still. But
interviewer has actually seen and talked to this person, has reported a written
p s t a l studies may also take time. We have to wait for the responses to come in,
set of real responses made at the time and there cannot have been too many
then send out two or three reminders at, say, fortnightly intervals, and extend
misunderstandings. Surely this is worth a whole lot more than little ticks in
the deadline to capture the final trickle of returns.
boxes returned by you-know-not-who!' This point of view is perhaps more an
Inherent in the use of interviewers is, or ought to be, the problem of briefing.
expression of research directors' anxiety at having become divorced from their
One Thursday afternoon a group of interviewers who had for many years been
subject material than a serious statement about validity. For interview results
part of a widely dispersed national field-force conducting consumer surveys,
may be genuine and rich, yet they can also be biased and unreliable.
were shown the following typical question from one of their own recent
Perhaps the most important determinant both of response rate and of the
schedules: 'In the past week, which of the following goods have you bought?'
quality of the responses is the subject's motivation. Data collection may be
followed by a list. The interviewers were then asked: 'What is meant here by the
regarded as a transaction in which it is usually fairly obvious that the research
phrase "in the past week"?' The first one replied: 'Well, today is Thursday, so I
worker stands to gain, in some sense, while the respondent is asked to give time,
would ask from last Thursday till today.' The second one disagreed with this:
thought, privacy and effort; anything that will make this transaction less
'The respondent might not yet have done her shopping for today,' she said, 'so
unequal and one-sided will help both the rate of returns and the quality of the
I would ask her from last Wednesday until yesterday.' The third interviewer
responses. Thus it is important for the interviewer to be able to explain how the
pointed out that a week had seven days and that Sunday was not usually a
particular respondent came to be selected for the sample and why it is important
shopping day, so she would naturally ask 'from the preceding Tuesday'. The
that he or she, rather than someone else, takes part. Extrinsic motivators such
fourth interviewer said that 'in the past week' meant just the current week on
as a preliminary postcard, a promise of a reward, the payment of 'expenses' or a
whatever day the interview took place, so she would always ask 'starting from
gift of a 'free sample', may sometimes also be effective in persuading people to
the previous Monday', and the results for the whole country would be ironed
participate. Rewards intrinsic to the subject of the survey, such as parental
out statistically. The last interviewer said that she had been with the company
interest in the health of their children or the belief that important changes will
a long time and knew what they really wanted, and so she always asked
be made as a direct result of the survey, can also improve results very markedly
respondents to tell her what they had bought 'on your last big Saturday
-there may even be a danger of over-eagerness creating a bias. One of the most
shopping expedition'.
important parts of interviewer training and briefing is to develop instant ways
This example shows, first of all, that even such a simple word as 'week' can
of engaging the respondent's interest and attention, to create and sustain
have quite different meanings for different respondents and interviewers.
'rapport' at just the right level, and to leave the respondent feeling that
Second, that because we think that everyone else means the same thing when
something pleasant, interesting and worthwhile has been accomplished.
they use the same word, it does not occur to us that it may need an explanation
One more advantage of using interviewers may be mentioned: in some
or a qualification. Third, that it is necessary to conduct pilot work on even
studies, the interviewers are asked to make on-the-spot assessments; for
apparently trivial matters. Fourth, that every question schedule has to be
example, OF the state of repair of a dwelling, of the proximity of the nearest bus-
discussed with the interviewers, tried out by them, followed by further
stop. While once again raising the issue of the reliability of such assessments, it
discussion and a firm briefing. And finally, that some of these problems might
is clear that they can best be obtained by means of a personal home visit.
not have arisen if efforts had been made to maintain better contact with the
interviewers.
Disadvantages of interviews
The disadvantages of using interviewers are to some extent a reflection of their Ethical considerations
advantages. Obviously, interviewers are much more expensive than postal The basic ethical principle governing data collection is that no harm should come
questionnaires (which cost little more than the expense of paper and printing or
to the respondents as a result of their participation in the research. If, for
duplicating, the cost of two envelopes and two stamps per subject). The larger
example, a respondent has been upset by some of the questions in an interview,
or the more dispersed the sample, the greater the total cost of the interviewing
then the interview may have to be abandoned rather than risk upsetting the
operation. Travel costs and call-backs add to this.
84 Questionnaire Design, Interviewing and Attitude Measurement Standardized inferuiews 85
respondent still further. It goes without saying that data collection should not be agency will conduct a proportion of check-interviews in every survey. To make
turned into an attempt to sell things to the respondent - not only is this tlus possible, respondents are usually asked for their name, address and phone
unethical but it gives social research generally a bad reputation. The respond- number, which will be treated as confidential.
ent's right to privacy and the right to refuse to answer certain questions, or to
be interviewed at all, should always be respected, and no undue pressure should
be brought to bear. In the case of children, permission should always be sought The process of responding
from their parents, teachers, club leaders or whoever is in charge of them at the
time. Let us now look a little more closely at what we are asking our respondents to
do when we ask them a question. What is going on in their minds? How do they
formulate their responses? Suppose the interview is a standardized one and that
The interviewer with a clipboard the interviewer has just spoken the words: 'When did you last read a copy of
Punch magazine?' Let us assume that the purpose of the study as a whole has
What actually happens during a standardized interview? The situation during a been explained satisfactorily to the respondent, that a certain rapport has been
structured, standardized interview is radically different from that of a depth set up between the two parties to the interview in reasonably congenial
interview, and yet some of the same rules and cautions apply. For a start, the surroundings, so that the respondent is at ease and is perfectly willing to try to
interviewer in a standardized interview is almost invariably a woman; this is answer questions. Let us further assume that the interviewer has spoken the
presumably because in many societies potential respondents, especially women, question correctly (reading errors do occur at times) and without particular
find it easier to be approached by a woman than by a man, are more likely to emphasis, and has previously ascertained that Punch magazine is known to the
establish rapport with her, ask her into the home and answer questions more respondent. We must also hope that the question will be heard and understood
frankly. Besides, survey agencies usually find that it is cheaper to take on women correctly by the respondent.
and that women can more readily be employed part time or on an hourly or daily A search process of some complexity must now begin in the respondent's
rate. It has also been argued that, on the whole, women are more empathic and mind. First, a search specification or mental template must be formed: the essential
generally make better interviewers. meaning of the question is formulated as an instruction to the brain to start
The interviewer-with-the-clipboard finds herself in a far less comfortable searching. At this point there are many possibilities that could cause unreliabil-
situation than the depth interviewer. Gone is the quiet office atmosphere, her ity: some respondents may, for example, interpret the word 'read' in the above
respondents will not have been found for her by a recruiter, there will be few question as 'read text and cartoons from cover to cover', some as 'idly leaf
consoling offers of refreshments (or else too many, this time pressed upon the through it in a waiting room' and some as 'buy a copy'. Search templates will be
interviewer by hospitable respondents) and generally she will be worlang formulated accordingly, obviously with very different results. The search now
against the clock. The schedule-bound interviewer is on her own, out in the proceeds, through 'the attic of the mind', trying to find the relevant occasion on
streets or at the bus station in all weathers or looking for some elusive address which Punch was 'read' and to date it. This process is likely to become very
in an unfamiliar neighbourhood. She has to carry her own equipment and find inaccurate unless the occasion is in the recent past or it can be placed accurately
and button-hole her own respondents, or else locate a hard-to-find address only in relation to some other event, for example 'last week-end; I know, because my
to meet with refusal or perhaps with no response at all (in which case one or mother-in-law came to stay with us and she brought a copy with her that she
more call-backs may be required). had read on the trainJ. Otherwise, the respondent will continue to t h n k back.
Interviewing can be interesting and rewarding (though it is usually poorly The search will be terminated at a fairly early point because long pauses in an
paid), but it can also be morale-sapping, very tiring physically and mentally, and interview are difficult to sustain, and because the respondent may come to feel
sometimes dangerous. Not every respondent will have fascinating things to say, that there 'must be' a quick and simple answer to this question. Eventually some
and interviews can be extremely tedious and repetitive. Some people are very rather nebulous recollection may be 'made to fit' the template: 'I probably saw a
rude or unpleasant, while others may voice provocative opinions, yet the copy at the dentist's; that was, well, let me see, how long ago? I know I should go
interviewer must remain outwardly calm, pleasant and friendly, try to sustain every six months, so let's say: about three months ago, that should be near
good rapport and never show in her expression or behaviour what her own enough right.'
values are. Sometimes she will find it hard to maintain her composure and Having half-formulated an answer, this must now be passed through a
'correct' social distance when the interview reveals circumstances that cry out number of filters. Was there a more recent occasion? Can anybody check up and
for social justice. find out you were inaccurate? Will the interviewer laugh at you if you admit that
Small wonder, perhaps, that there is a high rate of turnover in many interview you do not read Punch? Is there anything self-incriminating in seeing it but not
forces. It is also said that, after a time, some interviewers become 'burnt out': buying it yourself, or not going to the dentist more often, and so on? Having
they are no longer interested, no longer conscientious, they are only doing it for passed these various censoring filters the response now has to be formulated in
the money. For this reason, and of course to detect outright cheating, a reputable terms of words that make a sentence, leaving out the ruminative 'hmmmm's'
86 Questionnaire Design, Inferviewing and Aftifude Measuremenf
interview as too robotically standardized. It is part of the interviewer's job to take
and 'ah, well let me see's' and such apparently irrelevant details as the visit to the prsonal and situational variables into account, though not to the extent that
dentist. In this particular case, there also has to be a subtraction process: three they will constitute a source of bias. Any measure (see Chapter 9) will contain
months ago was when? And so, in the fullness of time, the words 'it was last two kinds of errors: random errors, which we should seek to reduce for the sake
June' are spoken, hopefully heard and understood correctly by the interviewer of greater accuracy but which would not bias the results in one direction or
and field-coded with a tick against the 'within the current quarter' box. If this another; and sysfematicerrors, which are much more worrying and which would
whole process of responding seems potentially fraught with error, then let us cause the final, accumulated results to be an indistinguishable mixture of 'trueJ
ask ourselves how much worse it might be under less-than-ideal conditions, answers and 'interviewer effects' or interviewer bias. Examples of systematic
dealing with a non-factual topic (for example'do you think we do enough for the errors might be habitual attempts to hurry less well-educated respondents;
mentally ill these days?') and with an embarrassed or resistant respondent. regularly offering a one-sided explanation of a particular question; or failing to
hand show cards to more 'intelligent-looking' respondents. Random errors. due
. --
to carelessness, inaccuracies, misunderstandings and the like, will cause our
The interviewefls central task results to be distributed around an imaginary mid-point which constitutes the
'true' finding, rather like attempts to hit a bulls-eye. Much more dangerous are
As we have seen earlier, the job of the standardized interview is that of dafa systematic errors or biases, since these would result in a substantial over- or
I under-estimate of the 'true'value, thus perhaps forecasting the wrong results in
collection; at this point in the study the interviewer does not have the luxury of an election, exaggerating or minimizing ethnic differences, or over- or
using her own initiative by adding questions to the schedule or following up underestimating the prevalence of domestic violence. Because such systematic
interesting ideas. She is bound by the requirement of stimulus equiualence, by the errors or biases generally go undetected, we cannot make allowances for them;
notion that every respondent should understand a given question in the same we donot even know whether they have caused our findings to be too high, or
way as every other respondent. Deviation from the exact wording of the too low, or are present only in part of our sample and to what degree. Our key
questions, from their pre-determined order, from the manipulation of prompt effort must therefore be directed towards the prevention or reduction of hidden,
cards or from an even tone of voice can all play havoc with this process. The idea sysfernafic bias in the interviews.
that we can ever actually obtain true stimulus equivalence is of course How is this to be done? On first thought, the answer would seem to lie in
unrealistic, but it is something to be aimed at in principle because interviewer absolutely rigid standardization of the interviewers' behaviour and of the
bias is probably the largest source of error in survey work of all kinds. To put it interview schedule. Every interviewer training course, every briefing and every
another way: ideally, when we look at our statistical results and find, say, that interview schedule should strive for precision and conformity in conducting an
significantly more men than women said'yes' to a given question, we want to be interview, so that all interviewers will do 'exactly the same thing'. However, this
sure that we are dealing here with a genuine sex difference and not also with presents us with a problem because what we are seeking to produce is not
some greater or lesser degree of 'interviewer effect'. In short, we must strive 'stimulus equality' but 'stimulus equivalence' (see above, page 86); that is, that
I towards obtaining data that are 'uncontaminated' by the interviewing process.
the respondent's understanding of the question or task will be the same as that
I T h s aim has been attacked as wrong-headed in principle as well as
of every other respondent interviewed by this or any other interviewer in the
unattainable in practice, and as denying the 'reality of the situation' while survey. This means that interviewers must also be trained to use their
indulging in a positivist fallacy and misrepresenting the basically interactive judgement in departing sometimes, and within certain limits, from the exact text
character of the data being collected. These critics suggest that we are on a self- of their schedules in order to offer an explanation, or some more familiar re-
deceptive quest when we seek to apply 'data-collection tools' to human beings in wording, to a respondent who does not comprehend, or has misunderstood the
order to obtain some pseudo-objective responses. They would see the situation question - while always trying to remain non-directive and not 'putting the
as essentially subjective and interactive: any 'answer' can only be understood if answer in the respondent's mouth'. This is not easy, and it is a test of the
we know how the respondent felt about the interviewer, about being interviewer's skill to do this smoothly yet without causing a systematic bias.
interviewed, about the topic and the particular way it was perceived, and about In their interviewer training programmes, some agencies make a distinction
the social and hierarchical relationship between interviewer and respondent at between three types of questions: (a)facfual questions, where the interviewer is
that particular moment. Our attempts at creating a standardized situation expected to read out the question as printed on the schedule but has some
would be regarded as 'scientizing' and falsifying the information being collected 1 .. r
I latitude to offer explanations, correct misunderstandings or offer a previously
# "
which is essentially interactive in nature. The unit-of-analysis should not be agreed definition; (b) attifude and opinion questions, which interviewers are
II
1 1
some 'answer' to a question but rather a social episode, a culturally determined r I . . .
forbidden to explain or reword in any way; and (c) certain standard classi\ication
dialogue between two strangers playing roles. questions (for example, on household composition), where the interviewers are
I
While svmpathetic to these concerns, to accommodate them would require an expected to devise their own probes to ensure that they have correctly obtained
entirely diffeient type of research process to which our present techniques are all the information required. This goes a long way towards resolving the
poorly suited. On the other hand, we should not think of the structured
88 Questionnaire Design, Interviewing and Attitude Measurement Standardized inferviews 89
contradictions between the requirement for standardization and the need for compensate for the shortcomings of the questionnaire designer. The phrase 'a
government that people can believe in'may, or may not, have the same meaning
flexibility and 'equivalence'.
Let us take a simple factual question from a survey on behalf of the Notown as 'a credible government' to every one of our respondents; hence we may, or
private bus company. On the interview schedule, the question reads: 'Did you may not, have resolved the equality=equivalence dilemma. Ideally, attitude
make a bus journey yesterday?' This question is not as easy as it looks. Some questions and statements should be so well designed and so fully piloted that
they can be read out and responded to without any variations, explanations or
respondents may say 'well, my firm sends a coach to pick us up, does that count?'
Or 'do you mean a Notown bus or a municipal one?' or 'no, but I did get a lift substitutions. In this instance, rather than offer a substitution at the briefing
from my brother-in-law; he drives his mates to work in his van'. Or 'I'm a the word 'credible' should have been recognized as problematic at the pilot stage
pensioner, and the ambulance came to take me to the day centre'. Or 'yes, I and firmly excluded.
always go into town on a Friday to do my shopping' (evidently has not realized Let us- now summarize the position we have reached, as far as the
that the question refers to Sunday, 'today' being Monday). Indeed, if the standardized interviewer's central task is concerned. Since respondents, ques-
interview does take place on a Monday, then the schedule will probably contain tions and situations differ so much, it would be unrealistic to demand of the
interviewer that she restricts herself rigidly and exclusively to reading out the
an instruction to ask about the preceding Saturday -which complicates matters
still further! What is the interviewer to do in such situations? Of course, the questions on her schedule in the prescribed manner. This would not produce
question should have been more adequately piloted so that a better phrasing 'stimulus equivalence' and would also make it difficult to sustain good rapport;
that is, a friendly willingness on the part of the respondent to answer further
could have been devised, and she may have been briefed in respect of some of
questions. What we require of the interviewer, and what good question
these queries. Also, a well-conducted survey would have instituted controls for
the day of the week. But the interviewer should also be given the necessary phrasing and good briefing should help her to produce, is that each of her
respondents should have an adequate understanding of every question and that
background knowledge about the purpose of the study and of this particular
she achieves this aim by non-directive means.
question. For example, she should be aware that the reason for the question is
As we have seen, it is good practice to distinguish between different types of
that the Notown Bus Company wants to know its present market share; that is,
questions. Factual questions should be read out as printed, but - within strict
what proportion of all bus journeys (defined, for example, as 'fare-paying week-
limits - the interviewer is allowed to explain, to correct misunderstandings and
day journeys by licensed public carrier) take place on its vehicles. On no account
to provide definitions. Atfitudinal questions (see Chapters 8 and 11)are much
should the interviewer therefore be tempted, when answering the respondent,
more sensitive, and no deviations should be allowed. In the case of standard
to say 'well, all we need to know is: did you catch a Notown bus yesterday?'
classification questions the interviewer can be given more latitude in order to
which would be grossly 'leading' and would yield a falsely high estimate of the
obtain all the relevant information.
Notown market share. Nor should she re-word the question by saying:'Yes, but
However, while she is working in the standardized mode, the interviewer
yesterday, did you go to work on a local bus, a Notown one or a municipal one,
maybe?' which provides unwanted prompts, emphasises journey-to-work and should not be expected to compensate for inadequacies in question design and
pilot work. Thus, the problem of ensuring equivalence really starts long before
puts 'yesterday' at the beginning, any or all of which may produce systematic
the field intemiews, with the work of the questionnaire designer.
biases. Her repetitions and clarifications must be non-direcfioe ones, but this is
more easily said than done, and only training, good briefing and experience will
enable her to satisfy the requirement that all respondents should understand the
question in the same way, without bias and without having had an answer 'put The interviewel's principles of performance
in their mouth'.
These problems get much worse when we are dealing with attitudinal questions,
If the above already seems like a tall order, there is more to come. Every
fieldwork agency and every briefing manual will have its own guidelines, which
for example 'in your view, could Party Z produce a credible government?' or 'Do
will cover many additional points about the interviewer's task and the problems
you think we are doing enough to help people in poorer countries?' Respondents
she may encounter. Here are some examples:
may ask the interviewer for clarification: what is the question getting at? But
attitude questions are notoriously sensitive to wording and nuance problems,
and so the interviewer is not allowed to expand on the question. She might well
try something like 'Well, what's your opinion? What do you think? In your view,
This elusive quality, which keeps the respondent motivated and interested in
could Party Z produce a credible government?' (without emphasizing 'could'). If
answering the questions truthfully, is one of the main reasons for employing
the respondent asks for the meaning of 'credible', the interviewer may first try
interviewers. There can be too much or too little rapport, and either would be
'Well, whatever it means to you', then perhaps offer Well, you know, a
undesirable. If the interviewer creates too little rapport, or fails to sustain good
government that has credibility'. It is just possible that an alternative has been
rapport, then the respondent will be resistant, will start guessing or joking, will
agreed at the briefing such as: 'It means a government that people can believe in', not take the task seriously and will try to cut the interview short. If rapport is too
but by now we are in deep water. We should not expect the interviewers to
90 Questionnaire Design, Interviewing and Attitude Measurement Standardized interviews 9 1
schedule. Some probes will be quite general, non-directive and non-specific; for
good, then the interviewer will find herself cast in the role of a helper or a social example, 'could you tell me a little more, please?' or 'any other points you'd like
worker or a personal friend, the interview schedule may have to be abandoned to make?' Other probes might be more specific; for example (after a No to the
half-way through, and the interview becomes unduly prolonged and difficult to bus journey question): 'Could you tell me why that was?'. Still others may
terminate. require several follow-up questions, and sometimes interviewers are instructed
The interviewer must play a role, and must obtain the respondent's co- to probe for specific issues - costs, timing, uncomfortable seats, waiting
operation in playing the role of 'good respondent', so that the role-play of a periods. It is best, for the sake of standardization, to print such additional
standardized interview can be performed. The interviewer must a t all times questions on the interview schedule rather than leaving them to be phrased by
remain detached and professional in her attitude yet be relaxed and friendly; she the interviewer.
must offer identification about herself and her organization but she must not Here is an example of a question with probes:
get personally involved; she must promise confidentiality,she must explain how
the respondent came to be selected and allay any misgivings the respondent may
have about letting her into the home. Yet she must also appear open, friendly How would you say that this area differs from the others around it? (PROBE:
and trustworthy, not 'cold' or 'official', and she must avoid exerting pressure on WHAT KIND OF AREA IS IT? ARE THE PEOPLE ANY DIFFERENT?)
respondents or arousing the thought that she may be trying to sell them (WRITE IN VERBATIM)
something. During the interview she must avoid answering personal questions
such as 'well, what would you say?' or 'do you vote for the X-party, too?' or 'don't
you agree with me that. . .', so as not to bias the responses, though she may
promise to have 'a general chat' afterwards. She will have to deal tactfully with
personal comments or approaches from male respondents, yet maintain good
rapport. Some respondents regard the interview situation as a licence to
complain about everything that is wrong with the world, others are just The interviewer is expected to write down verbatim on the interview
naturally very slow and discursive, while still others are unresponsive, taciturn schedule everything the respondent says (which may not be an easy task) for
or too brisk. - later coding and analysis. Unfortunately, it has been shown that some
With these, and with dozens of other problems, the interviewer must try to interviewers have strong expectations about what respondents 'really mean' or
find a way of coping that will neither offend nor bias the respondents, while 'are trying to say', and so they 'hear' (and write down) things that were never
keeping them 'in role' in a good-humoured way and maintaining rapport at just said. For this reason, special attention should be paid to recording problems
the right level. At the same time she must deal unobtrusively but efficiently with during interviewer training.
her recording tasks, especially the verbatim taking down of answers to open- Probes are one of the main advantages which interviews have over postal
ended questions. Of course she must also thank respondents at the end of the questionnaires, but they are also one of the most serious sources of interviewer
interview and promise those who ask for the results of the survey that her bias. Research has shown that many interviewers tend to be selective in their
research organization will send them an outline of the findings in due course. note-taking and often get into faulty habits which introduce a systematic error.
social class, ethnicity, age and educational background - in short, she should poor management of show cards;
aim at respectable social neutrality. If she needs help in creating the right 'image', biased probes;
the agency will be able to assist her. asking questions out of sequence;
unreliable field coding;
biased recording of verbatim answers;
RESPONDENT
SELECTION
poor management of 'problem' respondents;
Sampling (see Chapter 3) is very much part of the interviewer's job. She will inadequate management of situational problems or of
have been given instructions for the selection of her respondents, and these will 'special procedures'.
generally be of two kinds: either she is given a series of addresses, names-and-
To these must be added the imperfections and biases that may emanate from the
addresses or workplace locations; or she is given a quota, for example fifteen
structure and wording of the interviewing schedule itself; these will be dealt
working-class married women aged between thirty-five and fifty, not in a full-
with in other chapters.
time occupation. The addresses will have been chosen for her from existing
records; for example, a professional body listing, or a follow-up sample of
discharged bankrupts. Sometimes she will have to interview all adults living at Telephone interviewing
a given address, at other times she will be instructed to make a choice according
to certain criteria. She will be given further instructions about what to do in The most obvious advantage of conducting structured interviews by telephone
cases of refusal (substitution rules) or if there is no one at home (call-backrules). is their low cost. It has been estimated that face-to-face interviewers spend only
Departure from such instructions, for whatever reason, will impair the quality about one-third of their time in conducting interviews, the remainder of their
of the sample and hence affect the results of the research. time being taken up by travel and by locating respondents. Besides savings in
In the case of quota sampling the interviewer usually has more latitude. Most travel expenses and travelling time, telephone interviews also offer savings in
interviewers know from experience where, in their neighbourhood, they are other ways. They are generally conducted at a noticeably faster pace than face-
most likely to find respondents who meet the day's requirements, but therein lie to-face interviews, and permit close and continuous supervision at a central
risks of bias, too: a preference for railway station forecourts may undersample facility. The results of a set of telephone interviews are usually available in much
the car- and bus-commuters and the stay-at-homes; a preference for depart- shorter time, while substitutions for refusals and not-at-homes can also be made
ment store exits may over-sample shoppers, and so on. Since they are under very quickly. Speed is therefore a second major advantage of telephone
considerable time pressure, interviewers may sometimes stretch the limits of interviewing - an important factor in fields such as advertising, media research
their quota categories, for example to include an otherwise co-operative and opinion polling. However, computer-assisted face-to-face interviews, using
respondent who is probably just too old or too young; or they may be tempted lap-top computers which can transmit their data nightly by telephone to a
to include unmarried co-habitors in the 'married' category. Once again, any central processing computer, can offer turn-round times which are very
departure from instructions will cause error and may cause a systematic bias in competitive.
the selection of the sample. The most obvious draw-back of telephone interviewing is the problem of
sample representativeness, especially in countries where telephone ownership is
well below the 80-90 per cent found in some Western countries. This is not
To sum up: main causes of bias always the major handicap it might appear to be. Many studies do not require
nationally representative samples: for instance, studies of professionals, of
We have looked at many potential sources of systematic error that attach to the industrial managers or of commercial firms can safely assume that all their
standardized interviewing method of data collection and with which the sample members will be accessible by telephone. Sometimes it is reasonable to
interviewer must be trained to deal. While each source may overlap with others, define the sample as representative not of all school teacherslwar veterans1
we have noted the following categories: private pilots but only of those who are accessible by telephone.
Given that not all households or individuals are accessible by phone (note that
Before the interview: we are concerned here with access, not with telephone ownership), in what
departures from the sampling instructions, by the interviewer; ways will our sample be biased? Not being accessible by telephone is usually
adverse impression management by the interviewer. linked with lower income, with being young and male, and with having moved
During the interview: house recently. In addition, some subscribers choose to go ex-directory (in
poor maintenance of rapport; Britain the proportion is currently some 25 per cent of all subscribers and rising),
rephrasing of attitude questions; but these are unlikely to fit the above categories. On the other hand, in rural
altering factual questions; areas a widely distributed population can be sampled more easily, and perhaps
careless prompting; more accurately, by telephone.
98 Questionnaire Design, Interviewing and Attitude Measurement Standardized inferviews 99
In principle, there are three ways of generating a random sample of telephone In addition to conventional interviewer skills the telephone interviewer will
numbers: (a) from some other sampling frame, such as the computerized post- obviously need an excellent telephone manner, especially when speaking to
code address file (PAF), from a membership list, or from an existing set of those who resent having their privacy invaded by a stranger, who fear for their
records, in which case a phone number will have to be found for each name. (In personal safety, or who think the interviewer is trying to sell them something.
Britain, telephone numbers are, in practice, likely to be found for about two- Special skills will also be needed to make do without prompt cards and when
thirds of such a sample.); (b)by drawing a random sample from the most recent requesting an interview with a member of the household other than the person
telephone directories themselves (this procedure will miss out ex-directory answering the telephone. But in most ways the problems of telephone
subscribers); and (c) by 'random digit dialling' (RDD), where a computer selects interviewing tend to be very similar to those that are encountered in structured
telephone numbers at random from within a set of stratified geographical interviews in general.
regions which correspond to telephone area codes. (This latter method has the
advantage of potentially including people who are ex-directory.) Since the same
private telephone often serves several people, a sample drawn by telephone Selected readings
number is often more like a sample of households than a sample of individuals.
So long as there is no country-wide, computer-accessible, comprehensive and Beed, Terence W. and Stimpson, Robert J., 1985, Sumey lntemiewing, Unwin Hyman,
up-to-date list of the names, addresses and (where appropriate) telephone London.
numbers for the entire adult population, all these methods of sampling will Fowler Jr, Floyd J. and Mangione, Thomas W., 1990, Sfandardized Survey Intemiewing, Sage,
remain relatively cumbersome - though probably no more so than other London.
probability sampling methods now in use. As we have noted, in telephone Gordon, R.L., 1980, lnfemiewing: Sfrategy, Techniques and Tactics, Dorsey Press, New York.
interviewing it is quick and easy to make substitutions for refusals and for
respondents who are not at home. Turner, Charles F., and Martin, Elizabeth (eds), 1984, Sumeying Subjecfive Phenomena,
What other biases might we expect with telephone interviewing? For Russell Sage Foundation, New York.
example, is the refusal rate higher, compared to conventional interviews? Phrasing questions in interviews.
Despite some misgivings, this is not usually the case, and the telephone refusal Sudman, Seymour, and Bradburn Norman M., 1974, Response Effects in Surveys, Aldine,
rate is sometimes actually lower, perhaps because of the advantage of Chicago.
'interviewer invisibility' to some respondents. But time of day is important: About interviewingand question writing, recall problems and interviewer-respond-
evening and week-end refusal rates can be high. For a longer interview it ent interaction.
sometimes helps to make a preliminary call, to book a time at which the Hyman, H.H., et al., 1954, lnfemiewing in Social Research, University of Chicago Press,
respondent can take an extended telephone call. In some countries there is Chicago.
increasing resistance against 'cold' telephone calling. Early experimental studes of interviewer bias.
What kinds of questions can be asked over the telephone? Conventional Kahn, R.L. and Cannell, C.F., 1961, The Dynamic of lntemiewing, Wiley, New York.
interview schedules have to be specially adapted for telephone interviewing, but An older text, very detailed on problems of bias and interviewer skills. Still useful.
otherwise all but the most complex kind of question can be asked successfully
over the phone, and the telephone often seems to reduce resistance to'sensitive'
items. Problems may, however, be experienced with respondents who are hard- TELEPHONE
INTERVIEWING
of-hearing, with elderly respondents and those from minority groups - but Frey, James H., 1989,Sumey Research by Telephone, Vol. 150, Sage Library of SocialResearch.
probably no more so than with conventional interviews. Each interview Very useful and practical, deals with sampling problems, computer-assisted
schedule will have to be piloted and adapted to telephone use and computer telephone interviewing and administrative aspects.
assistance as necessary.
Dillman, D.A., 1978, Mail and Telephone Sur~eyr,Wiley, New York.
Some agencies operate a mixed-method model, using both conventional and
telephone interviewing. This will require detailed research to show whether or Groves, Robert, Biemer, Paul, Lyberg, Lars, Massey, James, Nicholls, William, and
not the two different modes produce comparable data on similar respondents Waksberg, Joseph, 1988, Telephone Survey Methodology, Wiley, New York.
within the same study; otherwise it might not be valid to merge the results of the A wide-ranging description of telephone-interviewing applications.
two data sets. The mixed-method approach is also likely to reduce the speedy
turn-around advantage of telephone interviewing. Some agencies have set up
large pools of respondents who have agreed in advance to be interviewed by
telephone; bias may be reduced by structuring each sample to be interviewed
along demographic lines, while the interviewing of special sub-samples becomes
a ready possibility.
Questionnaire planning 101
local solutions for specific dilemmas, is through small-scale field trials. Guess- that since there is no interviewer, there can be no interviewer bias. In a sense,
work, intuition, expert advice and spurious orthodoxy are no substitutes for a ghost interviewer is still present because the respondent may conjure up an
properly conducted pilot work. image or a stereotype of the organization which sent the questionnaire and of
the kind of person who might be asking these questions. In other words, the
respondents will interact with the questionnaire and may 'project' some kind of
I. Main method of data collection person or organization 'behind' the questions, and this may bias their responses.
As we have seen, there are many methods of data collection in social research, SELF-ADMINISTERED
QUESTIONNAIRES
but here we are concerned primarily with those methods that use a 'question- The self-administered questionnaire is usually presented to the respondents by
naire' in the broadest sense, namely the standardized, formal interview; the an interviewer or by someone in an official position, such as a teacher or a
postal, self-administered questionnaire; and the group-administered question- hospital receptionist. The purpose of the inquiry is explained, and then the
naire. Each has advantages and disadvantages, and our final choice will depend respondent is left alone to complete the questionnaire, which will be picked up
on its appropriateness to our purpose and to the means at our disposal. We later. This method of data collection ensures a high response rate, accurate
should also bear in mind some of the issues that have already been raised in sampling and a minimum of interviewer bias, while permitting interviewer
Chapter 6 on standardized interviewing techniques. assessments, providing necessary explanations (but not the interpretation of
questions) and giving the benefit of a degree of personal contact. Research
MAILQUESTIONNAIRES VERSUS STANDARDIZED INTERVIEWS workers may in this way utilize the help of someone in an official capacity who
is not a skilled interviewer. However, the greatest care is needed in briefing such
The main advantages of postal questionnaires are: persons or they may, with the best intentions, introduce fatal biases.
(a) low cost of data collection;
(b) low cost of processing;
(c) avoidance of interviewer bias (but see below); The group-administered questionnaire is also largely self-explanatory and is
(d) ability to reach respondents who live at widely dispersed addresses or given to groups of respondents assembled together, such as school children or
abroad. invited audiences. Depending on the size of the group and its level of literacy,
two or more persons will see to the administration of the questionnaires, give
The main disadoantages of the postal questionnaire are: help where needed (in a nondirective way), check finished questionnaires for
completeness and so on. Sometimes, variations in procedure may be introduced.
(a) generally low response rates, and consequent biases; For instance, the audience may be provided with empty booklets; some slides or
(b) unsuitability for respondents of poor literacy; for the visually handi- a film may be shown; and then a group of numbered questions might be read
capped, the very old or for children below the age of, say, ten; often aloud, one at a time, while the respondents write their answers in the booklets
unsuitable for people with language ddficulties; next to the question numbers. This ensures that all respondents answer the
(c) no opportunity to correct misunderstandings or to probe, or to offer questions in the same order and that they all have the same amount of time to
explanations or help; do so. Groups of forty can readily be controlled in this way, but contamination
(d) no control over the order in which questions are answered, no check on (through copying, talking or asking questions) is a constant danger.
incomplete responses, incomplete questionnaires or the passing on of
questionnaires to others;
2. Approach to respondents
(e) no opportunity to collect ratings or assessments based on observation.
Most of the issues under this heading have already been touched upon in
The advantages and disadvantages of the interview are almost a mirror-image Chapter 6 on standardized interviewing; basically, whether by interview or by
of these points. Interviews often have a higher response rate; they offer the mail, we must find ways of gaining the respondents' co-operation and of
opportunity to correct misunderstandings and to carry out observations and motivating them to respond to our questions. The ways in which we approach
ratings while controlling for incompleteness and for answering sequence; and respondents are therefore of paramount importance.
interviewers can often succeed with respondents who have reading or language
difficulties. But interviews are expensive and time-consuming to conduct and to
process, there are always the risks of interviewer bias, and interviews are usually
INCREASING
RESPONSE RATES
too expensive to reach a widely dispersed sample. The whole area concerning the 'best' way to approach respondents has long
Incidentally, in the case of a postal questionnaire it may not be altogether true been beset by myths and by controversy among practitioners. More recently
104 Questionnaire Design, Interviewing and Attitude Measurement Questionnaire planning 105
THE CONTENTS OF THIS FORM ARE ABSOLUTELY CONFIDENTIAL.
these problems have been the subject of a great deal of systematic research in a INFORMATION IDENTIFYING THE RESPONDENT WILL NOT BE
number of countries, and it is now possible to draw some tentative conclusions. DISCLOSED UNDER ANY CIRCUMSTANCES
However. these cannot be regarded as hard-and-fast rules and pilot work
remains essential. Reminders - mail surveys tend to suffer from low and slow response rates, so
The following factors have been found to increase response rates, either to mail reminders may be sent out when the rate of returns begins to drop, and perhaps
surveys or to personal interviews or to both: again after a week or two, to those who have not returned the questionnaire.
This becomes impossible if we have promised anonymity (see below), unless we
Advance warning - a letter or postcard informing the respondent of the study send out reminders to the entire sample, which might be too expensive.
in advance, and inviting participation.
Anonymity - this is not the same as confidentiality and can be promised only in
Explanation of selection - explaining the method of sampling used, and how the certain circumstances, for example to the members of a quota sample (though
respondent came to be chosen. even here names and addresses are sometimes requested so that a check
interview or phone call can be made). Respondents who have been sampled from
Sponsorship - this may take the form of an interviewer identification card plus a list of names and addresses are clearly not anonymous, though they can be
a pamphlet describing the research organization, or it may be a covering letter promised that any identifying information will be destroyed at the data-
or a letter of introduction from someone expected to be influential. Sponsorship processing stage. Identifying respondents by a code number rather than by
can be a powerful motivator (for example, if a postal questionnaire to members name may be reassuring to some extent. In the case of mail questionnaires,
of a trades union is accompanied by a letter of support from its general complete anonymity (so that returned questionnaires cannot be identified)may
secretary), but it can also go horribly wrong! (If, say, the general secretary is increase the response rate, but it then becomes impossible to send out reminders
thoroughly distrusted by a section of the membership; this would not merely to the non-respondents (unless we sent out reminders to the entire sample
reduce the response rate but would also introduce a powerful bias.) which might be too costly and will annoy those who have responded). Anonymity
is especially important in surveys that involve 'sensitive' topics.
Envelope - this refers to the appearance of the first envelope in a mail survey
(that is, not to the return envelope). It has a better chance of being opened and Appearance -there have beenpanv experiments with general layout, type face,
read if it is addressed to the respondent personally, if it has a stamp on it (that colour and quality of paper etc. in the case of postal questionnaires. No clear
is, not commercially franked) and if it looks professional' rather than like junk general conclusions have emerged; it is best to aim at a relatively 'conservative'
mail. The class of postage and any special delivery method seem to make little bGGpleasant appearance.
difference, though these effects may differ from country to country and from
time to time. Length - thickness of the question booklet, number of pages, and time required
to complete have all been investigated many times; generally these variables
Publicity - advance publicity in the local media can be helpful to some surveys, tend to interact with
provided it is favourable!Negative local publicity, or generally negative publicity
about all surveys can be very damaging to response rates. The topic. and its degree of interest to the respondent - long and complex
interviews or questionnaires will often be completed successfully if the topic is
of intrinsic interest to respondents (for example, if it is about their children), or
Incentives - small incentives have generally proved helpful, but larger incentives if they believe that their responses will have a direct influence on policy.
often are not. Future incentives, for example the chance to win a major prize if
the questionnaire is returned or the interview completed, are generally helpful. Rapport - this does not really apply to postal questionnaires. In respect of
Interviewers may also be more persistent if they know they can offer a reward, interviews, it is part of the interviewer's skill to establish and sustain rapport, so
thus reducing non-contacts. that the respondent remains motivated and will complete the interview (see
Chapter 6).
Confidentiality- all survey data must be treated as confidential, in the sense that
only the researcher(s) will have access to them, and steps must be taken to Return envelopes -it has often been alleged that non-respondents will steam the
ensure that no information d l be published about identifiable persons or stamps off return envelopes, thus reducing response rates, while others have
organizations without their permission (see also under ethical considerations, suggested that a 'real' stamp (rather than a business reply envelope) indicates
Chapter 6). Here we refer to an explicit statement or promise made to potential trust and will increase response rates. The evidence seems to favour the latter.
respondents in order to overcome possible apprehensions. For example, the The above determinants of response rates can be classified as (a) extrinsic (for
following may be displayed prominently on the front of the question schedule:
106 Questionnaire Design, Intemiewing and Attitude Measurement Questionnaire planning 107
example sponsorship; incentives) and (b)intrinsic (for example, the topic; length survey in the first place! In principle, therefore, we can often ascertain a bias in
of time required). Obviously there may be other factors that will influence our returns along demographic lines (such as those we have used to draw our
response rates. Among these are situational variables. Efforts have been made to initial sample), but this will not tell us much about a bias in the opinions or
get in touch with people who have been listed as 'refusals' or as 'non-contacts' by behaviour with which our survey is concerned.
field interviewers, in order to find out if these are a cause of systematic bias; that Sometimes we can try to do something more complex. We can undertake a
is, whether their unavailability or refusal to take part was in some way related multivariate analysis (see Chapter 15)of our returns and discover what are the
to the subject matter of the survey. If there was no such relationship, for instance strongest correlates of our dependent variable(s).If these correlates are variables
if the interview was refused because of illness or because the respondent was just for which we have information about our non-respondents (typically, these
going out, then these non-responses are classed as 'situational'. In such cases, an would again be the demographic variables which we have used to draw our
appointment for a call-back interview can often be successful. However, call- sample), then it might be possible to produce estimated values for our non-
backs are expensive and have diminishing rates of return. respondents, at least on some of our dependent variables. This is a hazardous
procedure, which might best be confined to populations about which we know
a good deal already, such as medical practitioners or school children. In other
DEALING
WITH NON-RESPONSE cases, it might be safer to do no more than to indicate the direction of the bias
Despite increasing sophistication in the approach to respondents, some steps due to non-response; that is, we might report that our results concerning
have to be taken to deal with non-response in any survey and especially in a newspaper reading among northerners are likely to be an under-estimate (or an
postal survey. These steps have to be anticipated and planned; it is simply not over-estimate) due to non-response problems - and leave it at that.
good enough to find, eventually, that only 38 per cent of the questionnaires have
been returned and then try to do something about it. First, let us note that a low ACCESS TO SCHOOL CHILDREN
response rate can usually be overcome without too much difficulty by sending
out some more questionnaires or by conducting some further interviews. The In recent years there has been an increase in the number of surveys and other
issue here concerns not the number or proportion of non-respondents, but the possibility of large-scale investigations among school children. The structure of the educa-
bias. We need to find out whether the reasons for the non-response are tional system provides a ready-made sampling frame in terms of age and often
somehow connected with the topic of our research; for example, in a general sex, in terms of educational level, geographical area and so on. If access can be
study of attitudes to unemployment, have many of the unemployed in our obtained to whole school classes, then data can be rapidly obtained by means of
sample been unwilling to respond? In other words, is there a confounding bias? self-administered questionnaires completed in the classroom; such procedures
To reduce this possibility, first, many steps can be taken to boost the over-all have obvious advantages over home visits or mail questionnaires, though self-
rate of response in our sample, as we have seen, and any such steps should be completion questionnaires for children are more difficult to design and require
tried out in the pilot work. Next, having done our best to maximize the response more intensive pilot work than questionnaires for adults.
rate, we have to attend to the question of bias. Some simple sociodemographic However, parents and schools are (like many organizations) very sensitive to
checks are often possible. For instance, if we have names and addresses for our the presence and activities of social researchers. School authorities have special
sample and for those who have replied or been interviewed, then we can run responsibilities to the children and their parents and, moreover, some schools in
some comparisons to see if, say, there are too few northerners among our recent years have been almost over-run by research students. It is important,
returns, or if people whose surnames begin with J or T are under-represented. therefore, to take particular care when we seek access to samples of children in
In other situations we may be able to find out if we have a disproportion by schools.
gender, or by age, in our returns. For a start, even though we may know some teachers or school heads in a
What is to be done about such a bias? Some survey practitioners would engage particular area, it is always best to write first to the director of education or the
in statistical weighting; that is, they would statistically boost the cells in the chief education officer of the appropriate municipal education authority. The
sample that are deficient due to low response rates. This, however, assumes that letter will seek permission for an approach to be made to the head teachers of the
within each cell the non-respondents have the same attributes or experiences as local schools in order to conduct research among their pupils (and staffs, if the
the respondents, so that the reasons for their non-response are purely latter are also to take part). Accompanying this formal request will be a
situational or at any rate are unconnected with the topic of the survey. But can statement which outlines the aims, design and methods of the research, and
we be sure of this? After all, our checks for non-response can usually be includes some justification for doing it, usually expressed in terms of its
conducted on the basis of only a few demographic variables and these may, or relevance to education, child health etc. This may well require brief reference to
may not, be associated with our topic of interest. Thus, in a study of newspaper- previous research in the field and to the researcher's experience, indicating his or
reading habits it might not matter that we have too few northerners among our her qualifications and status. Any measuring instruments to be employed
returns, provided that there is no association between newspaper-reading habits should
* .
be listed, especially if published tests are involved. The researcher must
and living in the north -but this may have been one reason for conducting the also be prepared to submit copies of questionnaires and texts for approval. When
108 Questionnaire Design, Infemiewing and AffifudeMeasurement Quesfionnaire planning 109
approvalin principle is given to a researcher to approach individual head teachers questions) tend to be very offputting to respondents. We must put ourselves in
(with whom a meeting is generally requested), an assurance of the confidential- their place. Having had the purpose of the study explained to them in a cover
ity of any data obtained is usually required. This confidentiality applies to the letter or by the interviewer, and having agreed to co-operate, they now expect
responses of individual children, the names of participating schools and often the some interesting questions dealing with the topic of the study; for example, their
education authority itself, none of whom should be identifiable in any published journey to work or their children's education. Instead, they are being steered in
report or thesis. Likewise, no data from individual children should be made a different, rather 'personal', direction by being asked a series of questions about
available to anyone outside the research team, not even to their parents or class their private life! To us such questions as 'are you singlelmarriedlliving as a
teachers. The exact terms of confidentialitywill be agreed with the officer of the couplelwidowedldivorced?~would be no more than routine classifying items,
education authority concerned. Finally, in the approach to authorities and to the but to the respondents they may well refer to highly sensitive aspects of their
schools, the timing and duration of a study must be considered and mentioned. lives. We have little enough 'right' to intrude on them as it is, and if we start off
The timing of the fieldwork in a research study is an important issue for the with questions of this kind we should not be surprised if we get no further
researcher; the schools, too, will want to know how much classroom time the response. Unless there are very good reasons to do otherwise, personal data
study will require. For example, a prolonged study will have to take account of questions should always come near the end of a questionnaire and should be
the structure of the school year, and the holidays between the terms, especially preceded by a short explanation such as 'now,-- tohglp us-classify
--- youranswers
in summer. Half-term holidays, school outings etc. also can upset or delay data a-nd to make-our statisticalcomparisons, would you mind telling us . . . how old
collection arrangements. School examinations are a constraining influence, for you are?' and so on.
however interesting or important a research study may be, examinations form Towards the end of the pilot work our questionnaire will consist of a series of
a vital stage in the education of adolescents, and head teachers will be most question modules or sequences, each concerned with a different variable. We
reluctant to upset the teaching of examination classes. After examinations, must now consider the order in which these modules should appear on the final
however, children are often much more readily available. It must be borne in questionnaire. In this, we shall want to bear two sets of considerations in mind:
mind, however, that with the end of the school session classes are often the internal logic of the inquiry, and the likely reactions of respondents - often
reorganized within a school, or children moved to other schools in the system so these will have conflicting requirements!
that a particular group of pupils involved in a study may no longer be available Consider, for example, a readership study. We might start with a set of
as a group. A more usual problem of data collection in schools is the absence of 'awareness' questions, which are designed to tell us what newspapers and
children due to illness or the loss of children who move away from an area magazines the respondent has seen or has heard of. These might be followed by
during a research study. This problem is less serious where school classes are some 'factual' questions, dealing with the respondents' own behaviour; for
large, consisting of over thirty children, so that adequate numbers of subjects example, what daily paper(s) they read (if any), what they read in the week-end
remain. But if children have been matched in pairs in the research design, the and so forth. Next, there might be a module on magazines and, after that, a
loss of one can mean the loss of both members of the pair from the study, since module on book reading and library borrowing. We hope that, so far, the
rematching to preserve numbers is often impossible. respondents have not found the questions too difficult and that co-operation is
In some schools the head will wish to obtain parental permission for the readily forthcoming. We might now offer a module on likes and dislikes in
research. The best approach in this case is to offer to draft a circular, to be signed newspaper contents; the next modules might ask about preferences for different
by the head, briefly explaining the research and giving the proposed starting content areas such as sports, stock-market news and so, eventually, we might
date. The children concerned can be asked to take such a circular home to their come to political attitudes and the political orientation of newspapers. By now,
parents. In the second part of the circular the parents will be told that their child we shall have moved also from simple question-and-answer sequences to check-
has been chosen (among others) to take part in the proposed research and that lists, ratings and perhaps to attitude statements. Finally, we shall need to ask
parental permission will be assumed unless the parents object before the proposed some personal data questions and come to the end with a 'thank you1.
starting date. Thus the onus is put on the parents to withdraw their child from We hope that, from the respondent's point of view, this has seemed an
the research, if they wish. interesting, sensible and non-threatening experience. Piloting the questionnaire
as a whole may, however, show us that some respondents become very
embarrassed by the 'awareness' questions at the beginning because they feel it
3. The build-up of question modules puts them 'on the spot', whereas they would not mind answering some
questions about their own reading behaviour right at the beginning. The pilot
work might further show that questions to do with book reading and library
Typically, the novice st%rt-tto deAgnna questionnaire by putting a rather borrowing cause difficulties because respondents tend to over-claim (prestige
forbiddine
- -- - - u set of auesXns at t G top of a blank sheet of paper. These are likely bias) - but then cannot remember the titles of the books or the names of the
to ask for game, ahdress, age, maritd status, number of children, religion and so authors. Perhaps it might be better to continue with more newspaper questions
on. However necessary, questions of this kind (classifying or personal data and save the library questions till last? Or, if people are irritated by these, would
110 Questionnaire Design, Interviewing and Attitude Measurement
Questionnaire planning 111
it be better to end with an easier module, say, about the part of the paper which WORKSHOP I: FUNNELLING QUESTIONS
they like best? Perhaps we should also give some more thought to the placement
of the political-attitude scales. Will they be influenced by the immediately Suppose we want to know whether some people avoid boiled sweets
preceding module dealing with the preferred sections of a newspaper? Will the because they are said to be harmful to the teeth. It would not do to ask
respondents be too tired by then? Should we try, among the personal data, to them a question such as, 'do you believe that boiled sweets are
ask for political party affiliation or preference? Or if not there, then where else harmful to the teeth?' or, 'do you avoid eating boiled sweets because
can we suitably ask about this? you feel that they harm the teeth?' These would be grossly leading
Meanwhile, how are we getting on with the balance of question types? Have we questions, and, besides, the respondent may never eat boiled sweets,
got too many open-ended questions, which are time-consuming and require a or may avoid them for some other reason. Obviously, it would be
lot of writing? Should we put all the pre-coded questions together at the valuable if we could get some of the respondentsto say spontaneously
beginning, irrespective of logic or content, so as to leave all the open-ended that they avoid boiled sweets because they are damaging to the teeth,
questions to the end? And should we perhaps treat the attitude scales separately before we suggest it to them and before they become aware of what
as a self-completion technique? the questions are really about. Therefore, we may start off with some
Consider now a somewhat more ambitious study, say, a survey of criminal very broad questions such as: 'What is your opinion of boiled sweets?'
victimization and perceptions of crime. We shall need to ask respondents about 'What do you think of people who eat boiled sweets?' Each question
any personal experiences of crime which they may have had - burglaries, car provides the respondent with an opportunity to mention the issue of
thefts, assaults, pickpocketing - but we shall also want to ask about their dental decay spontaneously. Next, we might ask more restricted
perceptions of crime in general and in their area, and about their attitudes to the questions, such as: 'Do you eat boiled sweets at all?' 'Did you eat
degree of seriousness of different crimes, as well as their attitudes to the police boiled sweets when you were a child?' 'Do you allow your children to
and to the courts. Which of these modules should come first, or last, and why? eat boiled sweets?' Each should be followed up with 'why is that?' if the
Crimes are often not reported to the local police, or there may be a problem with reply is negative, thus providing further opportunities for the dental
an insurance claim, or with ethnic prejudice, or two youths may fight over a girl, issue to emerge spontaneously. After that, we may narrow the
or a child may steal from its parents. These are very sensitive areas, and people's questions still further: 'Do you believe that boiled sweets can be
ideas about crime may not fit the appropriate legal categories, nor will their ideas harmful in any way?' 'What would happen if you ate too many boiled
about the relative severity of crimes necessarily correspond with the sentencing sweets?' 'What are some of the disadvantages of eating boiled
policies of the courts. Yet often people have extremely strong attitudes in sweets?' Note that the dental problem still has not been mentioned
respect of such matters as self-defence, child abuse, exceeding the speed limit, directly. Finally, we bring up the problem as nondirectively as
drugs and the behaviour of the police. To make the final decisions on the order possible: 'Some people say that eating boiled sweets is bad for your
in which modules dealing with each of these topics should appear on a question teeth, but others say that it makes no difference. How do you feel
schedule will not be easy. about this?' Or, 'Do you believe that eating boiled sweets is bad for
your teeth, or do you think that most people's teeth will not be
damaged by eating boiled sweets?' And so on. By proceeding in this
4. Order of questions within modules way we not only increase our chances of obtaining what we are
seeking through a spontaneous reply, we also place the whole issue of
boiled sweets and tooth decay in the context of some of the other
We now come to a consideration of the order of the questions within each factors that determine the eating of boiled sweets. This context can be
module. Let us assume that we have a detailed specification for our module and very important; it may well be that other reasons for not eating boiled
that we have some suitable questions from the pilot work, some'open' and some sweets are mentioned far more frequently than the possible danger to
'closed', with which we are trying to construct part of an interview schedule. In the teeth.
what sequence can we best approach the issue? Should we ask about attitudes
first, or begin by asking about personal experience and then 'broaden out'
towards more general attitudes? Or perhaps the other way around? A filter question is used to exclude some respondents from a particular question
The funnel approach, preceded by various 'filter' questions (see below), is one sequence if those questions are irrelevant to them. Thus, in the above example,
.- - -.. - - type of sequence which
well-known A
- -- ispften -- - - though for our particular
.-- .used
we might wish to ask for some factual information about sweet-buying
purpose some other 'sequence, suggested by further pilot work, may turn out to behaviour and boiled sweet purchases. Obviously, if the respondent never buys
be better. The funnel approach is so named because it starts-offthe4mcd_ulewith these sweets then there is no point in asking about frequency, weight, type of
aquestions
very
, - - . - - question,and thencprpgresivelynarrows down the s c o p
broad
until in the end it comesto some- very specific points.
- >
--- -e----
Xe shop, type of container, colour preferences and so forth. Therefore, our
112 Questionnaire Design, Interviewing and Attitude Measurement Questionnaire planning 113
illustrative question sequence will be preceded by a filter question such as 'do their thoughts roam freely, unencumbered by a prepared set of replies. We
you buy boiled sweets from time to time?' or, 'have you bought any boiled obtain their ideas in their own language, expressed spontaneously, and this
sweets within the past two weeks?' If the answer is negative, the interviewer spontaneity is often extremely worthwhile as a basis for new hypotheses. In an
will be instructed to skip the next few questions and proceed to the beginning of interview, however, there is the risk that we will obtain, not so much a rounded
the next question sequence (a routing instruction). and full account of the respondents' feelings, but rather just what happens to be
Each survey produces its own problems of question order, which makes it uppermost in their minds at the time. If this is true, then we may still ask
difficult to offer general principles. We try, as much as possible, to avoid putting whether what comes first to the respondents' mind is not also most important
ideas into the respondents' minds or to suggest that they should have attitudes for them, and for us.
when they have none. Therefore, with regard to any issue, we may want to start Free-response questions are often easv to
.,-..I--
ask, difficult to answer. and still
- - -.--- --
with open questions and only introduce more structured or pre-coded questions more diffi&l+to --.,.>-
aAalyse.
- As a rule, we employ a classifr~itionprocess known as
at a later stage. &ding (see Chapter 14,which requires drawing up some system of categories, a
Some researchers prefer to start each question sequence by asking a few coding frame. The design of such coding frames-aid the actual coding operation
factual, multiple-choice questions about the respondent's own habits, back- require trained staff and are extremelv time-consuming;; for this reason
ground or experience. Further questions follow naturally about the respond- researchers have to curb their desire to have too many open questions.
ent's attitudes concerning these points and about wider issues. For example, we Sometimes, if the first answer seems a little ambiguous or does not go far
may ask the respondents how often they go to church, if at all, whether they enough, we can instruct the interviewer to probe. This often takes the form of
went to Sunday school as a child, and then follow these questions with some asking the respondent to explain further or to give reasons for something stated
broader ones on religious issues and attitudes. Other researchers prefer to start earlier; at times, a particular issue may be brought into the discussion
each module with wide-open questions and ask about the respondent's own deliberately, if the respondent has not already mentioned it. Such probes should
behaviour or experience at the end. In both approaches there is the danger that be as nondirective as possible, thus: 'Could you say a little more about . . . ?"Why
inconsistencies and contradictions between attitudes and behaviour will emerge did you say just now that . . . ?"Now, what about the . . . ?"And how do you feel
which the respondent may try to cover up. about . . . ?' The risk of interviewer bias is probably at its highest whenever
Our final choice of approach and sequence must be determined by our own probes are employed. They are 'safe' only in the hands of the most highly trained
survey problems and by the results of the pilot work. and experienced fieldworkers, and many survey organizations avoid probes
altogether.
Closed questions can be attitudinal as well as factual. The alternatives offered
5. Question types are very much part of the question and should be reported as such, for they
guide the respondent's answers. Suppose that we ask the question: 'Some people
in this community have too much power. Who are they?' Left to their own
thoughts, some respondents might think in terms of political parties, others in
Broadly speaking, most questions are either 'open' or 'closed'. A closed question is terms of criminals and racketeers, still others in terms of certain professional
one in which the respondents are offered a choice of alternative replies. They groups. Consider the different effects of offering either of the following lists of
may be asked to tick or underline their chosen answer(s) in a written possible answers:
questionnaire, or the alternatives may be read aloud or shown to them on a
prompt card or a slide. Questions of this kind may offer simple alternatives such Blacks OR newspaper owners
as Yes and No, or the names of five political parties in an election; or they can Asians landlords
offer something more complex, such as a choice of ways of keeping order in a Arabs the police
classroom or a choice of motives for smolung cigarettes. Ch'inese big businessmen
Open or free-response questions are not followed by any kind of choice, and
the answers have to be recorded in full. In the case of a written questionnaire, Each of these leads the respondents in a particular direction, which may or may
the amount of space or the number of lines provided for the answer will partly not correspond with their own thoughts. Moreover, they are expected to
determine the length and fullness of the responses we obtain. Inevitably, some express an opinion; very few people would insist on a response such as 'none of
of this richness is lost when the answers are classified later, but it is useful to these'. Under these conditions, merely to report that X per cent of our sample
report a few such answers in full in the final report to give the reader some of stated that, say, landlords have too much power, would be grossly misleading
the flavour of the replies. Statistical tabulations are important and-miisiremain unless the other alternatives offered to the respondents were also reported. O n
our first aim, but they make dull reading. the other hand, the fact that we are directing the respondents'thoughts does not
The chief advantage of the open question is the freedom it gives to the of itself make the question invalid or worthless. We may be particularly
respondents. Once they have understood the intent of the question, they can let interested in comparing hostility toward landlords with hostility toward
114 Questionnaire Design, Interviewing and Attitude Measurement Questionnaire planning 115
newspaper owners, and the pilot work may have shown us that these two publications. (Diary techniques (see Chapter 13) may give the most valid and
groups are rarely mentioned in a free-response question. But we must be aware precise estimates, but they cause some respondents to engage in duty viewing or
of what we are doing and make our choice suit the requirements of our research. duty reading.) It is clear, therefore, that caution must be exercised when we
Incidentally, if we guide the respondents' thinking along particular lines in this compare prompted results with those of free-response questions.
way, it may also influence the answers to subsequent free-response questions
(contextual effects).
Closed questions are eager and quicker to- answer;
____ - - - - they -
.- _Crequire_no writing,
and quantification is straightforwarSThis often means that more questions can Let us now try to compare the open type of question with the pre-coded or
be asked within a given length of time and that more can be accomplished with closed one (see Tables 7.1 and 7.2). Each has its advantages and disadvantages,
a given sum of money. Disadvantages of closed questions are the loss of and most surveys will deploy a mixture of the two.
spontaneity and expressiveness - we shall never know what the respondents
said or thought of their own accord - and perhaps the introduction of bias by Table 7.1. Open questions
'forcing' them to choose between given alternatives and by making them focus
on alternatives that might not have occurred to them. Closed questions are
oftencruder andless subtle than openones, although this is not necessarily so, Advantages Disadvantages
and we d6lbse the opportunity to probe. There may also be some loss of rapport,
if respondents becomeirritated because they feel that the choice of answers fails Freedom and spontaneity of the answers Time-consuming
to do justice to their own idtias, so wh-t to i n c l u j ~ a n'Ot&.pl?e specify)' Opportunity to probe In interviews: costly of interviewer time
category.
Sometimes there may be good reasons for a s k i ~ m o both n in Useful for testing hypotheses about ideas Coding: very costly and slow to process,
OF-n and in ~ l o s e d f ~ ~ m ~ F o rif~we n ask,
c e ,'what are some of the thingZKat or awareness and may be unreliable
maG&rGnn&ove up in the world?' we shall get a pretty clear idea of the way Demand more effort from respondents
in which the respondent thinks the social system works and the relative
importance of several avenues of mobility, such as education, hard work, money,
luck. We get a free, spontaneous sketch in the respondents' own language and Table 7.2. Closed questions
containing their own ideas. This is most valuable, but it makes it difficult to
compare one group of respondents with another. Also, we cannot be sure that Advantages Disadvantages
such an impromptu sketch really contains all the factors that are important to
the respondent. A momentary lapse, a feeling of reticence, or the inability to put
ideas into words can cause the omission of significant points. Therefore, later in Require little time Loss of spontaneous responses
the interview we may ask the same question again, but this time we will produce No extended writing Bias in answer categories
a list which respondents may be asked to rate or rank (see Chapter 9),or from Low costs Sometimes too crude
which they may be asked to choose the three most important factors. Having
already obtained the spontaneous responses, there can now be little harm in Easy to process May irritate respondents
introducing a set of ideas obtained in the pilot work, even though some of these Make group comparisons easy
might not have occurred to some of our respondents. By using a 'closed' Useful for testing specific hypotheses
approach we ensure that the results of several groups can readily be compared
and that all respondents have considered the same universe of content before Less interviewer training
giving their replies.
W k i t y p e of cpestion &ves m-ad il results? When it comes to an
enumeratGn-o"T~tGSms of behaviour, such as newspapers or magazines read the FIELDCODING VERSUS OFFICE CODING
previous week or programmes watched on television, the open type of question
produces a lower yield than the prompt-list kind. This is partly due to temporary Field coding is one of the functions of the interviewer (see Chapter 6). A
forgetfulness and partly to the fact that some papers or programmes have low question is read out to the respondent as an open question, and the answers are
memorability.Weeklies and monthlies are not recollected as well as dailies, when coded on the spot by the interviewer, by ticking or circling among the categories
open and closed questions are compared. On the other hand, multiple-choice provided on the interview schedule (but these categories are not read out to the
questions are subject to a small degree of spurious inflation; people confuse last respondent). In a simple case, the respondent may be asked for country of birth;
week with some earlier week, or confound the names of programmes or the interviewer listens to the answer, and then ticks the appropriate category,
116 Questionnaire Design, Intemiewing and Attitude Measurement Questionnaire planning 117
questionnaire reaches the office and is coded, for the number of coding
for example, Latin America, South West Asia, Eastern Europe and so on. This categories cannot be unlimited. At a still later stage, during the statistical
technique may cause considerable error, bias and loss of information, and needs analysis, more information loss may occur if coding categories have to be
to be used with discretion. combined or when over-all scores or indices are calculated. Therefore, if we
Note that this is not the same as asking a closed question: the respondent is allow the respondent a free answer, we must ask ourselves at what stage the
not aware of the pre-coded categories which have been provided for the inevitable loss of some information will cause the least bias: during the interview
interviewer. Note also that the response itself is not recorded - it is lost, it itself, by asking for field coding; or later, if we ask for complete answers to be
cannot be retrieved for checlang or correction, or for re-coding later. We are recorded first. The interviewer in the field generally has a better appreciation of
forced to rely completely on the skill and training of the interviewer, who must what the respondent was trying to say, but the rushed circumstances of an
make instantaneous decisions (while preparing to ask the next question). It is interview may make it more difficult to fit the response immediately into the
easy to understand how errors, omissions and misclassificationsmay arise in the appropriate category. The coder in the office can ponder more carefully, but by
urgency of the moment, all the more so if the question demands a discursive then only the written record is available. The content of the question and the
answer (such as asking respondents what they think of keeping pets) - and the complexity of the answers will largely determine our decision, but the quality of
schedule contains a complex and subtle list of field-coding categories, such as our field force and of our coders will also have to be considered.
'make the house dirty', 'teaches children to take care of others', 'animals are A further point in connection with loss of information is the question of
better friends than people', 'too expensive', 'always good company, someone to relevance. Many surveys, regrettably, contain redundant or irrelevant ques-
talk to when lonely', all derived from the pilot work. Faced with such a very tions, which have been put in for the sake of 'interest'but have no bearing on the
general question, most respondents will offer a considerable amount of problems at issue. Also inevitable are the many open questions that will - due
comment, and it will be up to the interviewer to select and condense what seems to the freedom they give the respondent - produce a good deal of information
most important, and tick or circle the most relevant categories. To do this well that is not really relevant to the inquiry. In such instances, irrelevant
requires a great deal of briefing and training, and some bias and loss of information may be eliminated as early as possible by means of closed questions
information cannot be avoided. or field coding. However, we must be careful not to omit information that may
By comparison, office coding (see Chapter 14) takes place under more tranquil be important as contextual or background material.
circumstances. The responses will have been recorded in writing, the coding
These comments about loss of information apply also to the use of closed or
frame will have been derived in part from the hypotheses and the pilot work, and multiple-choice questions, with the added problem that the alternatives offered
in part from a sample of responses obtained in the main fieldwork, so the may bias the responses. All closed questions should start their careers as open
categories are likely to be more appropriate. Any doubts or queries can be ones, except those where certain alternatives are the only ones possible: the
discussed with the coding supervisor; if necessary, re-coding can take place names of the candidates in an election, the daily newspapers on sale and so on.
because the original responses are still available. Reliability checks can be Careful pilot work (see Chapter 4) is very important here; only by trying out an
conducted as desired. On the other hand, office coding is much more time- open question in various forms and attempting to 'close' it afterwards, can we
consuming than field coding, requires additional staff and facilities and adds gain an appreciation of the loss of information involved. Also, the pilot work will
greatly to the cost of the study. provide us with a set of multiple choices that will really fit the range of answers
to be expected and that will reduce, if not eliminate, the loss of information.
LOSS OF INFORMATION Whether we choose to put the question in open or closed form will partly be
decided by whether it is possible to provide a set of suitable multiple-choice
The problem of loss of information, of condensation and compression, should be answers. Surprisingly, the pilot work sometimes offers the possibility of closing
placed in perspective. The question is not how we can avoid loss of information quite subtle questions, thus avoiding the expense and possible distortion of
but rather at what point we can best afford to lose information. At first sight, it coding.
may seem as if the open question causes the least loss. All the interviewer has to
do is to ask the question and write down the reply. But already some loss of
possibly relevant information has occurred: the facial expression and tone of Selected readings
voice of the respondent, the hesitations and reformulations, the little digressions
and repetitions, not to mention large parts of a longer answer. All these will not Sudman, Seymour and Bradburn, Norman M., 1983, Asking Questions, Jossey-Bass, San
be recorded, for most interviewers cannot use shorthand, and most interview Francisco.
schedules provide only a limited amount of space in which to write. Inevitably, A detailed, practical discussion of factual and attitudinal question design.
even when attempting to get a complete record of an answer, the interviewer Hoinville, Gerald and Jowell, Roger, 1978, Sumey Research Practice, Heinemann, London.
selects, and selection may reflect bias. Sometimes, she only hears what she
expects or wants to hear. Nor is the interview situation the only point at which Converse, Jean M. and Presser, Stanley, 1988, Sumey Quesfions, Sage, London.
Some excellent practical advice from two experienced survey researchers.
loss or distortion of information may occur. Further loss will take place when the
118 Questionnaire Design, Intemiewing and Attitude Measurement
Schwarz, N. and Strack, F., 1991, Context Effects in Attitude Sumeys: Applying Cognitive Theory
in Social Research, European Review of Social Psychology, Vol. 2, Wiley, Chichester.
Moss, L. and Goldstein, H., 1979, The Recall Method in Social Sumeys, Studies in Education
No. 9 (new series), University of London Institute of Education, London.
1. Yes
2. No
around. It fails to offer any kind of middle position such as 'uncertain', casting different kinds of bait at different depths, without knowing what is
or 'it depends'. Nor does it have a 'no answer' or 'not applicable' (N/A) going on beneath the surface! The function of a question in an interview
position to cater for people who have had no experience of football schedule or questionnaire is to elicit a particular communication. We hope that
matches (though some of them might, nevertheless, have opinions on our respondents have certain information, ideas or attitudes on the subject of
the matter). For 'people that go' it might be better to have 'people who our enquiry, and we want to get these from them with a minimum of distortion.
go' to football matches. The probe 'please comment on why' is also If it were possible to do this without asking them any questions and without the
ungrammatical and might have been worded less curtly. Questions respondent having to 'respond' that would be so much the better, for the
will sometimes need to be worded colloquially, but this should only be questions we ask, the possible misunderstandings they provoke in the respond-
done with discretion and checked in the pilot work. Poor grammar ent, the choice or phrasing of the answers, and the recording procedures all have
may irritate some respondents and cause them to treat the whole influences on the final result that we could well do without. Would that there
questionnaire with contempt. were ways of loolung into the minds of our respondents without having to ask
Let us also look at the layout of the question. For a start, there is no them any questions at all!
'?' after 'matches'. The probe at the end is located too far to the right However, for the time being we have to rely on some form of questioning,
and leaves no space for the requested comments (a few lines beneath with all its attendant risks of bias and distortion, and on the fallibility of the
the probe would help). But more important is the fact that no respondent's own internal search processes. We must always remember that,
instructions have been given to the respondent. If, say, the 'yes' however much we try to standardize our questioning procedures in order to
answer is chosen, then should it be ringed, or underlined, or given a create 'stimulus equivalence', there will always remain differences in the way in
'tick' on the right, or on the left; or should the numeral '1' be ringed, or wluch questions are put to each respondent, and these may (or may not) have
crossed - or what? Perhaps little response boxes should have been important influences on the results. Similarly, what is understood by the
provided? To the left or to the right? These may seem pernickety respondent and what starts off the 'process of responding' (see Chapter 6) will
criticisms, but failure to attend to such matters of layout can produce be different every time, while whatever is recorded or noted down from the
endless trouble at the data-processing stage and may cause much respondent's answer will be selected and possibly biased. While our primary
loss of information while the respondent, too, may at times be baffled. concern here is with question wording and design, we must not imagine that
Such matters of presentation, instruction and layout should never be once questions go into the field they will constitute an absolutely standardized
left to a typist. They require the systematic attention of an experienced set of stimuli; nor will the responses reach us in 'pure' form. We must always
researcher and should be part of the pilot work. bear in mind the difficulties the respondents may have in understanding the
We now come to the most serious deficiency of this question and question and in forming an 'inner picture'of their own answers or reactions, and
one that is not immediately apparent: what is the purpose of the the degree to which these may be affected by inner curbs on private or self-
question? The researcher who produced this question had been incriminating information, by poor rapport, by the wish to maintain a social
asked to design a simple, factual question to ascertain how often facade and by the response expectations which the question may suggest. In
respondentsattended football matches. Whatever we may think of the other words, it is difficult enough to obtain a relatively unbiased answer even
question we have just considered, in no way does it qualify as a factual from a willing and clear-headed respondent who has correctly understood what
question, nor will it allow us to infer how often the respondent goes to we are after, without making our task virtually impossible by setting off this
football matches! The question is, quite simply, inappropriate - it 'train of responding' on the wrong track through poor question wording.
does not meet its stated purpose. In a sense the problem of question wording is akin to a sampling process. We
This problem is frequently encountered in questionnaire research, are sampling a particular universe of content in the respondent's mind: say, his
especially when we are asked to evaluate someone else's question- or her attitude to Mexicans. We are not intending to obtain the whole of this
naire. The questions may all seem quite reasonable, but without some universe of content but only enough of it to enable us to outline its salient
knowledge of their purpose, that is of the research design and the features, general direction (positive or negative), its depth or intensity, and
questionnaire specification, we cannot state whether the questions perhaps the presence or absence of specific beliefs. Our questions must be
will meet their objectives or not. adequate for this sampling process; they must not be too one-sided, and they
must make it easy for the respondent to answer fully. This means, first of all,
that the focus and contents of the questions must be right; second, that the
Some people still design questions as if the process of interviewing or of filling wording must be suitable; and third, that the context, sequence and response
out a questionnaire were rather like unloading a ship, with every item of cargo categories (if any) must help the respondent without unintentionally biasing the
labelled according to its contents and marked with a specific destination, so that answers.
it can be lifted out of the hold and set down as and when required. In reality, Each question also has a covert function: to motivate the respondent to
questioning people is more like trying to catch a particularly elusive fish, by continue to co-operate. A question that strikes respondents as rude, abstruse or
122 Questionnaire Design, Interviewing and Attitude Measurement Question wording 123
inconsiderate may affect not only their reply to that particular question but also by appropriate questions. There should be continual discussion and reference-
their attitude to the next few questions and to the survey as a whole. We must back to the specification document.
strive, therefore, to meet respondents half-way and to give and maintain the
general feeling that they are being treated, not in an adversarial manner but WORKSHOP Ill: QUESTION CONTENT AND PURPOSE
with respect and consideration. We must never forget that the respondent is
doing us a favour by taking time and trouble to answer our questions. In As we have seen in the preceding chapter, we should have a
designing questionnaires it is not merely important for us also to look at things questionnaire plan or outline giving the sequence of all the major and
from the respondents' point of view; we must make them feel that we are doing minor issues with which we are concerned. We must now decide, for
SO. each issue in turn, how thoroughly we will 'sample' it. Here, for
This can be done both by avoiding defects and by creating positives. Thus, we instance, is a question from a survey dealing with dental services:
should avoid humiliating respondents, baffing them with terminology, patron-
izing them or making them feel in the wrong. Politeness helps: instead of
staccato phrases such as 'marital status', 'length of residence' and so on, we
should state such questions in full, not forgetting to say 'pleaseJfor 'would you
( Have you heard of fluoridation of water supplies?
Yes ..............................1
mind'. Some questions will benefit from an introductory sentence or two by way No ................................ 2
of explanation, but we must neither talk down to respondents, nor talk over Don't know .............. 3
their heads by using technical terms or abbreviations. Keeping in mind that we
shall probably be dealing with people of differing educational backgrounds, our If yesJ
questions should be clear to the less well educated while not seeming too vague How d o you feel about fluoridation of water supplies?
or silly to the university graduates. The use of slang or street terminology, and Do you approve? ............................. 4
the commission of deliberate grammatical errors in an effort to maintain Do you disapprove? ....................... 5
rapport, requires the greatest care if misunderstandings or ambiguities are to be Are you undecided? ....................... 6
avoided.
Another way of maintaining the respondents' co-operation is by making the Probe all answers
questionnaire and the answering process more attractive. In self-completion and Why do you say this?
mail questionnaires the layout, printing, choice of paper, spacing, answering
directions and so forth should all be carefully considered, and piloted wherever There were no other questions concerning fluoridation in the
possible. For example, the method of answering multiple-choice questions survey. Presumably the researchers concerned regarded this as an
should be consistent throughout: either circling, ticking or underlining, and always interesting but relatively unimportant issue on which they did not
either on the right or on the left. Open-ended questions, which require thought propose to spend a great deal of interviewing time. Or perhaps the
and writing on the part of the respondent, should be kept to a minimum. Since pilot work had shown that very few people had ever heard about
reading is so much quicker than writing, every effort should be made to save the fluoridation. At any rate, the survey team must have made some
respondent time and trouble by the appropriate use of lines, insets, different decisions. First, they decided not to leave out this issue. Second, they
fonts, colours, headings, boxes and so forth. decided to sample it only briefly. Third, they decided that one
It happens too often that, once the final draft of a questionnaire has been multiple-choice attitude question, preceded by a filter question, and
assembled, the general layout and appearance are left to a subordinate. This is a followed by a 'why' probe, would suffice. They must have been
short-sighted practice. The person in charge of the study should personally tempted to ask more questions, perhaps designed to find out what
review the over-all appearance and layout from the respondent's point of view information the respondent had on this subject. They might have
and smooth out any remaining rough edges. One good way of Going this, in considered the possibility of making the questions more specific, for
addition to pilot work, is to read the questions out aloud to see if they 'flow'. instance by asking about the fluoridation of 'your' water supply. They
Another hidden attribute of every question is its link with the conceptual did not assume that everyone would know what fluoridation is or
framework of the study. There should always be a clear answer to queries such would have a clearly defined attitude toward it, but they did not find it
as 'Why are we asking this question? What is it doing here? How do we intend necessary to offer an explanation of this technical term before asking
to use the responses?' In the course of much pilot work and revision it is only too questions about it. The questions were kept as short as possible, using
easy to lose sight of the purpose of some of our questions and - as we saw in mostly familiar words.
Workshop 11above - to be left with questions that no longer meet their aims or, Do we know why the question was asked at all? Very often problems
conversely and even more importantly, with aims that are no longer being met of question construction make us realize that we are not clear enough
124 Questionnaire Design, lnfemiewing and Aftifude Measuremeni Question wording 125
in our own minds as to what the questions are about. Very likely, for forget how circumscribed our own experience is, and we take for granted that
instance, in the example just given, the main interest centred upon the whatever the word means to us, it will mean the same to everyone else.
reasons against fluoridation. If, instead, the main issue had been the In a housing survey it was found that many respondents reported fewer
amount of information possessed by respondents about the effects of bedrooms than their houses actually contained. They simply did not think of a
fluoridation, different questions would have been asked. study, a playroom, a sewing room or a guest room as 'bedrooms' in the sense
Greater precision concerning the purpose of the questions will intended by the survey. Similarly, in a survey concerning coffee, many
sometimes make it easier to avoid ambiguity in question wording. respondents failed to mention powdered or 'instant' coffee because they did not
Consider, for instance, the meaning of the word 'read' in a factual think of these products as 'coffee'. The last two examples point, first of all, to the
question such as: 'Which magazines have you read in the past seven need for definitions - both for the sake of the researcher and to help the
days?' Should 'read' here be regarded as synonymous with 'bought'? respondent. 'Is the child afraid of one or more animals?' we may ask a sample of
Or with 'bought and also read'? Or with 'borrowed but read'? Or could mothers, and add, 'that is, ordinary domestic animals, not lions, tigers etc.' More
it include 'borrowed but only glanced at'? 'Picked up in a waiting subtly, both examples also point to differences in frame of reference that may
room'? What about 'borrowed in order to read just one article'? What exist between researcher and respondent. A typical problem word might be
about exchanges of magazines? Annual subscriptions? 'Just looked 'family', a term which&-widely differen~meanines-de~ending on_.___ frame
one's ___
at the pictures?' of reference. Or, to give another example, in a study of saving, insurance policies
It is obvious that we must stop thinking of the meaning of words in were oftennot mentioned: some people don't think of them as a form of saving.
dictionary terms and instead ask ourselves what we are trying to find We must not assume that people have the information that we seek. They
out. What are we going to do with the data when we have collected may be reluctant to admit that they do not know or, being anxious to please,
them? This will make it easier to decide which of many possible they may guess. We must avoid giving the impression that respondents ought
meanings of a word we wish to employ. For instance, if we were to know. Such questions as 'what was the name of the author?' (of a recently
concerned with magazine sales or with personal expenditure, we borrowed library book) and 'what brand was it?' (of consumer goods) would be
might alter the question to: 'Which magazines have you bought in the better worded: 'Can you remember the author's name?' and 'Can you remember
past seven days?' However, if we were concerned with exposure to what brand it was?' O r we may offer a face-saving phrase such as: 'For many
magazine advertising, a very broad definition of 'read1 would be people, life is so hectic nowadays that they often cannot remember small details
appropriate. It all depends on the purpose of the question. from one week to the next.' If in doubt, and especially if technical terms or
difficult words are part of the question, a filter question can be helpful. Recall is
often poor even on quite important issues. For instance, it has been shown that
parents are quite inaccurate in their recall of details about child-rearing practices
Factual questions and early developmental progress even if their children are only three years old,
and the parents were part of a special panel study. In consumer surveys, to go
The adjective 'factual' is here used to distinguish this type of question from back more than two or three days can be extremely misleading.
others, thought to be more difficult t o design, that may deal with knowledge, It is generally best to keep questions short -preferably not more than twenty
motives or attitudes. However, factual questions are just as likely to present the words. If more than two or three alternative answers are possible, a show card
research worker with awkward problems. Consider, for instance, the use of the should be offered on which the respondents can read the multiple choices; they
f
word 'bought' (see above, Workshop ID). Does this include buying or others?
Does it include buying on credit? Does it include things paid for by nd bought
should not be asked to keep too much in their minds. Every effort should be
made to use familiar words. Even so, some of the simplest words can become
problem words if they are too vague - words such as: you, abouf, all, any, few,
on behalf of someone else? Does it include presents?
Mealtimes are another example. In England, the word 'tea' as in the question: government, fair.
- If we off& the respondent a list or a set of multiple-choice answers, we run the
'When do you usually have tea?' refers to a light meal (which may or may not
risk that these may become subject to ordinal biases. For instance, when asked for
include tea as a beverage) taken at almost any time after about three-thirty in the
a numerical estimate, people tend to choose a figure near the average or near the
afternoon. Some familieshave'tea'in the mid-afternoon. Others refer to the main
middle of a series. In a list of opinions or ideas, those at the beginning and those
evening meal as 'tea'. Some give their children 'tea' when they come home from
at the end may have a greater drawing power. Even when only two possible
school and then give them'supper'later. In other families a meal given to children
answers are offered, there may be a tendency to choose the latter. To overcome
is called 'tea' but the same meal taken later by adults is called 'dinner'. Different
these problems, some investigators randomize the order so that each inter-
customs exist in different social classes and in different parts of the country. The
viewer or group of interviewers (or each subsample of respondents in a mail
institution of 'high tea' further complicates matters. To say, therefore, that
inquiry)has the answers in a different order. Others use the split-ballof technique
something happened'at tea-time', or to hear that product X is always taken with
in pilot work, dividing the sample into two or more equivalent parts and
one's tea, requires elucidation. When drawing up a questionnaire we tend to
126 Questionnaire Design, Interviewing and Attitude Measurement Question wording 127
Thus, for recall items including periodical behaviour, closed questions and
presenting each part with a different answer sequence which makes it possible checklists generally yield higher frequencies than open-ended questions,
to measure the ordinal bias and to make allowance for it. Though such especially so in postal questionnaires. Conversely, open-ended questions are
procedures are expensive and demanding, they should certainly be applied to generally more sensitive to social desirability bias, especially in an interview
questions which recur from one survey to another; only by drawing general situation.
conclusions from split-ballot and similar pretests in many investigations will we
be able to arrive at a more precise principle that will enable us to minimize these
biases. WORKSHOP IV: FACTUAL QUESTIONS
The problems of the dou ble-barrelled and double-negative questions are mentioned
by many writers. 'Have you suffered from headaches or sickness lately?' (Would Let us start with what may at first appear to be a simple, factual
a positive answer refer to headaches, to sickness or to both?) 'Would you rather question: the possession of a videotape recorder or VTR set. A first
not use a nonmedicated shampoo?' (What does a negative answer mean here?) attempt at the phrasing of this question might read:
'Do you know if the deadline has expired yet?' (A negative answer might mean
that the respondent does not know, or knows the deadline has not yet expired.)
Such confusing questions should be avoided.
Another type of question to avoid is the hypothetical one, whereby
1 Do you have a videotape recorder?
Can we lay down some rules? you live?' we should allow for the possibility that the respondent does not know,
or that there are legal complications (because a house sale is in progress or
It would be nice to think that question wording could be carried out in because the landlord has died recently). Similarly, in the case of an opinion
accordance with a short set of simple rules or principles. Many authors have question such as 'Do you think, on the whole, that the government is doing a
tried to do this and have, at best, succeeded only in stating the obvious or in good job?', a 'don't know' response makes good sense. It has been argued that
listing pitfalls to be avoided. But question design is also a creative process, some people give a 'don't know' response in order to avoid thinking or
emerging from brainstorming sessions, adaptation of earlier questionnaires, committing themselves; but do we really want to obtain 'forced'responses which
listening to the tapes of depth interviews or group discussions, and to the are virtually meaningless? Indeed, sometimes 'don't know' responses can be very
contributions of experienced interviewers and helpful pilot-work respondents. important. For example, before mounting a health education campaign dealing
However, here are some basic rules: with sickle-cell anaemia, should we not first find out how many people have
heard of it or know what it is?
Length -questions should not be too long; they should not contain sentences of
more than, say, twenty words. If a longer sentence is needed, then it should be Use simple words, avoid acronyms, abbreviafions,jargon and fechnical ferms or else
broken up, so that there will be several shorter sentences, each dealing with only explain them. Respondents who do not know what the terms mean may feel
one or two tasks or concepts. It is often helpful, especially when starting a new intimidated and may guess or pretend to know the answers. There is an old story
section of the questionnaire or interview, to have some introductory sentences about a survey dealing with the Metallic Metals Company; many respondents
which will give the respondent time to adjust and will slow down the had strong views on its labour relations, the quality of its products, the dangers
interviewer. For example: to the environment and so forth when, in fact, the company did not exist! You
may know what UNCTAD is or does, or the meaning of HIV, or BMW, or
Now I am going to show you a number of descriptions of acts which are crimes. For each benzodiazepines or an exhaust catalyst, but do not assume that your respond-
one 1want you to indicate how serious it is. Of course, in one sense all of these acts are ents do. When in doubt, use pilot work.
serious because they are all crimes, but they are not all equally serious. For example, most
people consider murder to be a more serious crime than shoplifting. What I want to know Beware the dangers of alternative usage - t l s applies to even the simplest words
is how serious you personally think each crime is, and not what the law says or how the which 'could not possibly be misunderstood'. As we have seen (see Chapter 6),
police or the courts would act, or what anyone else would say. You indicate this by giving there is much disagreement as to the meaning of the word 'week'. A child's age
each crime a score from one to eleven. For example . . . may be given as his age next birthday. To some people, the word 'dinner'indicates
an evening meal, but to others it means a cooked meal, as in 'Usually I give him
and so on, followed by thirty descriptions of crimes which the respondent has to his dinner at lunch-time'. Words such as 'read' or 'tea' have many meanings. In
rate according to seriousness. some nutritional surveys it was found necessary to send the interviewers out
with plastic models and pictures of certain foods or meals in order to avoid
Avoid double-barrelled questions - 'Do you own a bicycle or a motorbike?'; 'Do ambiguities.
you think it is a good idea for children to learn German and Spanish at school?'
In such instances, the respondents are left in a quandary if they want to say 'yes' Some words are noforiousfor their ambiguity and are best avoided or else defined;
to one part of the question but 'no' to the other and, if they say 'yes', we do not for example, the words= and'have' as in'do you have a car?'Is this a question
know whether they mean 'yes' to both, or to one (and if so, which one). of ownership or of use or availability? What if the respondent has the use of the
family car but does not own it? Who is this ' ~ o d- the respondent, the
Avoid proverbs and other popular sayings, especially when measuring attitudes, household, the entire family or the firm?,And what if the respodent has access
for such sayings tend to provoke unthinking agreement. Instead, try to make the to two cars? Other notoriously ambiguous words are 'family', 'bought' and
respondent think afresh about the issue by putting it in other words. 'neighbourhood' - every survey agency will have its o s G t of dangerous
words and associated horror stories.
Avoid double negatives - for example 'Cremation of the dead should not be
allowed', followed by ageeldisagree. Here, the 'disagree' response is hkely to All closed questions should start their lives as open ones so that the answer categories
create difficulties because respondents who favour cremation have to engage in will be based as much as possible on pilot work (see Chapter 4). Even then,
a double negative (that is, they do not agree that cremation should not be allowed). always consider the inclusion of an 'Other (please specify)' category.
Try to put the issue in a positive way.
Beware 'leadingfquesfions- 'When did you last borrow a video tape?' It assumes
Don'f Know and Not Applicable categories are too often left out, both in factual (a)that all respondents have access to a videotape player; (b) that they generally
and in non-factual questions. If we ask 'Who is the owner of the home in which borrow tapes; and (c) that it is not done by someone else on their behalf.
130 Questionnaire Design, Interviewing and Attitude Measurement Question wording 131
Similarly, 'how often do you take a foreign holiday?', or 'do you generally buy the Did you go to the dentist/cinerna/etc. yesterday?
brand that is least damaging to the environment?' This raises a behaviour sampling problem. We want to end up with some
statistic about the typical frequency of this kind of behaviour. Are we assuming
Beware loaded words such as democratic, black, free, healthy, natural, regular, that, if we had a large and representative sample, the percentage we obtain
unfaithful, modern. would be typical of this kind of behaviour? Not, surely, if all our respondents
were interviewed on the same day, for different days of the week probably
Don't over-tax the respondents' memories by asking questions such as 'in the past would show a different pattern.
thirteen weeks, which of the following branded goods have you purchased?' A Shall we ask:
period of thirteen weeks may be thought of as a 'quarter' by planners and
economists but it is meaningless to most respondents and, in any case, far too Have you been to the dentist/cinema etc. within the past week?
long for the recall of relatively trivial items of shopping. For recurrent items of
behaviour it is risky to go back more than a few days. For more important events We have already seen how different the definitions of the word'week'can be; we
such as major acquisitions, periods in hospital, or being mugged, the respondent now ask: how do we know that this past week is representative of all the weeks
has first to be taken through the preceding few months with the aid of in the year? Perhaps we happen to be interviewing during a holiday period or in
'anchoring dates' such as public holidays, family birthdays, school holidays and the coldest part of winter. So should we then extend the period, and ask over the
the like, followed by an attempt to locate the event(s)in question in an individual past fortnightlthe past quarter? That produces problems of recall inaccuracies.
time frame. It would be better if we could stick, perhaps, to one week but spread the
interviewing period over the best part of a year to allow for seasonal variations,
Pay due attention to detail such as layout, show cards and prompts (note especially though this may not be practical.
the danger of overlapping categories, as with 'age 25-30', '30-%I), the wording Another approach is often used:
of probes (follow-on questions), routing or skip-to instructions and even
question numbering. Your data processors will not thank you if, for example, When did you last . . . ?
you start every new section of the questionnaire with question (I)! If the behaviour is at all infrequent, this obviously creates memory problems. It
also assumes that the 'last' occasion was somehow typical of habitual behaviour
It is worth reiterating that questionnaires are not created by following rules patterns so that we can arrive at a meaningful average; but is this true? On a
such as the ones above. Each questionnaire is unique (even if it is mostly mail questionnaire this question is also open to anchoring problems: responses
composed of questions that have been asked before but which are now being such as 'I remember, it was the day after our neighbour's dog got run over' or
applied in a new context) and presents its own special problems. The way to 'when my brother and his wife came to stay with us' - which cannot be
check on the wording of questions and, where necessary, to make improve- converted into days or weeks. (The alternative to this wording, 'how long ago is
ments, is through the pilot work. In the later stages of the pilot work, if doubts it since you last . . . ?' requires, in effect, two tasks of the respondents: fixing the
still persist regarding the best way to word a particular question, we may need occasion and doing a subtraction. Even if they get the former right, they may err
to use the split ballot technique: we obtain two or more comparable samples and in the latter respect.)
administer our question(s) in alternative forms to see the effects of different Or shall we dispense with all these time (and behaviour) sampling problems
forms of wording (see page 61). and simply ask:
How often do you usually. . . ?
Special problems
Here we have asked the respondents to do the time sampling for us. We shall
never know how far back in time they have gone in order to do their calculations,
QUESTIONS
ON PERIODICAL BEHAVIOUR how many exceptions they allowed themselves before they recognized that a
Typical of the many areas that can cause difficulties is the problem of measuring change in a habitual pattern had taken place, and what the word 'usually' means
items of recurrent behaviour such as going to church, drinking whisky, visiting the to them.
cinema or going to the dentist. We shall pass over problems of definition of Yet another procedure that is sometimes used is to request respondents to
terms, boasting, and the vagaries of memory, and concentrate purely on the keep a diary of particular kinds of behaviour over a period of, say, a fortnight.
study of periodical behaviour sampling. We could start by asking: Apart from the expense and the problems of low response and delay in filling-
in, we also find that people then start to engage more often in the behaviour they
are asked to note, in order 'to have something to write'.
So we see that it is really not at all easy to 'get at' such 'simple, factual' material
132 Questionnaire Design, Intemiewing and Attitude Measurement Question wording 133
-even with co-operative respondents, on unembarrassing topics, where a 'true' Electrician. He cuts the light off of people who do not pay the bill.
answer can be thought to exist and to be known to the respondent. Imagine the A steel erector. My father erects steel and iron for buildings, factories and coal
difficulties involved in finding out from a rather resistant wife what her
husband's pay is - a touchy topic to which she may not even know the 'true'
I
I
mines. He mostly works a lot of feet up.
answer! (See Workshop V, p. 133.) District manager of a brewery. He looks after a certain number of public
houses; he sees the right kind of people go into them and settles any disputes
CLASSIFICATION
QUESTIONS which may arise.
Classification questions are a special type of factual question that asks about age, Barber. He cuts men's hair and small boys too and big boys.
sex, marital status, income, education, occupation, family size and so on. These
a r e of special importance in stratifying the sample and tend to recur from survey Wives, on the other hand will sometimes introduce prestige distortion,
to survey; for both these reasons they are worth special attention. As we have describing the husband as an 'engineer' when he is a garage mechanic, as a 'civil
already seen, unless there is some special reason to do otherwise (for instance, servant' when he is a street cleaner, or as 'manager' when he is a stall holder.
in quota sampling), such questions should come right at the end of the Similarly, some husbands may exaggerate or trivialize the importance of their
questionnaire, by which time we can hope to have convinced the respondent wives' job.
that the inquiry is genuine. Even then, we ought to use an introductory phrase, Occupational classification systems can present many difficulties. How, for
such as: 'now, to help us classify your answers statistically, may I ask you a few example, should students be classified?What should be done about actors, some
questions about yourself and your family?' With these questions, too, we must of whom may be more famous and prestigious than others? O r about farmers
be particularly rigorous in excluding unnecessary detail. As mentioned before, with different acreages? And how should a married couple be jointly classified
there is no need to ask for the respondent's exact age, when all we really want when both partners are in full-time occupations that are classified differently?
to know is whether he or she is over or under forty-five. Or, to take another Also, in many countries such classification systems fail to cater for women. If
typical question, do we really need to know whether the respondent is separated, they are married and running the home, they are often classified according to
divorced or widowed? And must we go into quite so much detail with regard to their husbands' job, and if they are in full-time work, the classification schemes
household composition? often fail to provide for their occupations.
One need hardly stress that in gathering this kind of information, in In general, classifying questions tend to be predominantly sociodemographic,
particular, definitions are important, both for the interviewer and for the but for some inquiries a psychological classification would be more relevant -
respondent. There must be clear instructions concerning stepchildren, mori- for example, a score on a general-knowledge test, a do-it-yourself score, a
gaged house ownership, part-time occupations and so forth. romanticism score, a radicalism score or a frustration-resistance score.
The purpose of asking for the occupation of the main breadwinner is usually
in order to grade it on some scale of social class or social prestige. People often
give vague job identifications such as: clerk, engineer, businessman, farmer,
secretary, civil servant; these terms require further information before they can
be classified. One can ask supplementary questions or probe in vayious ways, but
the information needed for each job is different. We have found, however, that
the following two questions keep the 'unclassifiables' to well under 10per cent:
I I WORKSHOP V: SOME FIRST ATTEMPTS AT WORDING A
CLASSIFICATION QUESTION
Below are listed ten attempts made some years ago by a group of
(I)'What is the name of your job?' (2)'Please describe as carefully as possible the British graduates to phrase a factual question, asking married women
work that you do.' Moreover, even in written questionnaires directed to about their husbands' income. When reading these attempts, several
husbands (about their wives' jobs) or to schoolchildren (about their fathers' points stand out. First, most of the questions are too blunt: there
occupations),enough information will usually be obtained to make classification seems to be far too little awareness that this is a sensitive area, which
possible. may require a carefully worded introductory sentence or two, as well
It has been suggested that wives and children often do not know what their as some explanation of the reasons for asking the question. Second,
husbands or fathers do for a living. Nowadays, in Western societies, this is fairly it soon becomes obvious that the term 'income' is too vague and that
rare and likely to be due to special circumitances. (Questions about earnings, the data will not be suitable for comparisons unless we specify
though, are apt to be far less successful!) Even children of below-average whether, say, tax and other items have been deducted or not. Third,
intelligence can give job descriptions that are very clear and free of humbug. the types or sources of income need to be considered: weekly wages/
Here are some examples: annual dividends/pension/freelance professional earnings? In turn,
this suggests that the question needs to take account of wide
A constable in the M borough police. My father sees that shop doors are closed variations in people's economic circumstances.
at night and gives advice to passing motorists.
134 Questionnaire Design, lnhrviewing and Attitude Measurement
I Question wording 135
A. Is your husband salary, pensions and interest earned from bonds, shares and
- employed bank deposits
- unemployed No income
What is your husband's average gross income per week?
What is your husband's average net income per week? less than £3,000
9. Does your husband tell you what his income is?
How much does your husband earn
(a) each week?
(b) in a year?
Where did you get that information from?
C. Do you know what your legal husband's full income is?
If so, do you know approx how much tax he pays?
If so, do you know approximately how many other deductions
he pays from his income, i.e. life insurance, house mortgages £20,000 or more
Do you know if he has other sources of income besides his
job(s), i.e. share dividends, interest? G. Are you married? . . . . . . Yes/No
i
If yes, does your husband have a source of income?
D. Housewives have difficulties in deciding how well their hus-
Yes/No
bands are paid. Please tick the figure nearest to your husband's If yes, could you indicate how much your husband's income is
income and also tick whether you think it is satisfactory or not each week
0 - 3,999 . . . . . . . . . Pounds
4,000- 5,999 H. Is your husband in receipt of an income? (irrespectiveof source)
6,000 - 7,999 £ per year Yes/No
8,000- 9,999 If 'yes', do you know the amount of that income?
10,000 - 11,999 Into which category would you place your husband's income?
(a) up to £3,000 p.a.
(b) £3,001-£6,000 p.a.
~
I
100,000 - 100,499
Satisfactory Unsatisfactory
(c)
(d)
(e)
£6,001-£9,000 p.a.
£9,001-£I 2,000 p.a.
£12,001-£I 5,000 p.a.
E. Do you know how much your husband earns per week? (f) more than £15,001 p.a.
Yes No (please specify to within the nearest thousand pounds)
If your answer to question one was 'yes', could you now tick the Can you be more specific about the actual amount of your
appropriate box husband's income?
over £200 p.w. J. Does your husbandearn
less than £2,999 p.a.
£110-1 99 p.w. 0 £3,000-£5,999
£85-109 P.W. £6,000-£8,999
£9,000-£11,999
1 £60-84 p.w. 0 £12,000-£14,999
£15,000-£18,999
More than £19,000?
I less than £25 p.w.
K. How much money does your husband earn each month
F. Please mark one box to show the annual income earned by your (use the preceding 12 months)
husband for 19.. . When calculating income include wages/
136 Questionnaire Design, Intemiewing and Attitude Measurement
I Question wording 137
disadvantage that, if the survey extends over many months, the
zero C] 1,001E-1,5002
twelve-month periods will not be comparable.)
30E-150E over 1,500E None of the attempts really caters for the 'don't know' respondent,
nor for the wife who only has an approximate idea; whenever we
design a set of pre-coded responses we should always consider
including a 'don't know' category. Routing instructions (for example 'if
Yes, go to Question 29') are generally poor or absent, and not many
attempts include precise answering instructions. Thus, in the case of
'YedNo', what is the respondent supposed to do: ring? tick? cross
out? underline? This can lead to confusion and unreliability when we
The different question writers have different sets of assumptions come to use the data. Boxes are sometimes offered on the left,
and expectations, which are expressed in the ways they have worded sometimes on the right, if at all. The approach of B and D is much
their questions. These, in turn, force us to the most important point of gentler to the respondent than the bluntness of J. Evidently, the range
all: why are we asking this question? What kinds of detail or degrees of expected annual incomes differs widely (up to more than E100,000
of precision are required? What use will be made of these data in the a year!), while there are even preferences for the location of the E sign
statistical analysis? If, for example, the purpose of the question is (see K).
merely to divide a large sample into some three or four income
categories, then it may suffice to offer the respondents a pre-coded
question, using the limits of those income categories. But then again, LEADINGQUESTIONS AND LOADED WORDS
the question may be the first in a series of questions concerned with
savings and insurance, for which a good deal of detail may be needed Leading questions are so worded that they suggest what the answer should be
about financial circumstances and income sources. As we have said or indicate the questioner's own point of view. Examples might be: 'You haven?
before, question design has to be integrated with the survey's , forgotten to brush your shoes today, have you?"Most people nowadays believe
objectives and with the planned analysis. Pilot work will show how well in racial integration; do you?' 'Are you against giving too much power to the
the questions meet their aims. trade unions?' Generally, factual questions are less sensitive to 'leading' than
Several other points of difference are worth commenting on, some attitudinal questions.
of which you may have noted already. These students have clearly A 'loaded' word or phrase is one which is emotionally coloured and suggests
been warned against the dangers of having overlapping categories an automatic feeling of approval or disapproval: for instance Nazi, bosses,
(for example £60-90 pw; £90-120 pw - the sum of £90 pw could go interference, motherly, starvation, strike-breakers, British, intelligent, Socialist.
into either category) and gaps in category ranges (for example £60-90 Respondents are reacting not so much to the issue posed by the question as to
pw followed by E l 00-1 20 pw); both of these are mistakes frequently the loaded phrase itself. Split-ballot trials will show considerable differences in
made by beginners. Such gaps or overlaps would be obvious in the distribution of answers to a question using the phrase 'bosses' compared to
numerical categories such as we have in this example, but would be one using 'leaders' -both are loaded phrases but loaded in different directions.
far less obvious (yet equally troublesome) when we offer categories Once we are aware of the ~roblem,we can exercise our ingenuity in trying to
of, say, favourite types of reading matter. Example C, with its legalistic find a less loaded term, but the trouble is that we often fail to realize the danger.
emphasis, nevertheless fails to ascertain any actual amount! Several Words and phrases that are more or less neutral in one context or to one group
attempts seem not to distinguish between income and earnings, and of people may be highly loaded in another context or to another group, for
I
one (F) includes in 'earned income' those sources which are usually instance the term 'birth control'.
regarded as 'unearned' - this would obviously have to be sorted out Here are some items from a questionnaire dealing with various aspects of job
and respondents would need to be instructed in writing, or else the choice. Each item consisted of a pair of choices, only one of which could be
interviewers would have to be briefed on this point. Some attempts (B, selected by the respondent; this technique sometimes helps to overcome
D) ask additional questions which may, or may not, be relevant. problems of loading. In the pilot work, strenuous efforts were made to reduce or
Husbands with irregular earnings (for example shopkeepers, profes- eliminate loaded phrases:
sionals, salesmen) are poorly catered for, while the period covered by (I) A job in which you do a lot of OR A job where, once you have learned it,
the question (which week, month or year?) is usually left indetermi- hard thnking you always know how to do it
nate. (But note that F specifies a particular year and in a way so does
K - though very few women could answer either question readily,
even if their husbands are on regular incomes. K has the added
1 (2) Where somebody helps you
to decide what is to be done
OR Where you have to decide for yourself
how to do things
138 Questionnaire Design, Inferuiewing and Atfitude Measurement Question wording 139
(3) Where the pay is not very OR Where the pay is high, but you don't get would seem likely. They claim that they buy new tyres for their car when in fact
high, but you get a pension a pension when you retire they buy retreads; they deny reading certain Sunday newspapers of dubious
when you retire repute; the clothes that they buy are expensive; they would seem to make
donations to charity with great frequency and to visit museums almost every
(4) Where you work with your OR Where you work with your hands week! There is no simple answer to this problem, but, in addition to being aware
brain that it exists, there are two general measures that may help. First, use filter
questions, or word the main question in such a way that a low-prestige answer
The results showed that the items were, in fact, fairly evenly balanced in pairs is equally possible; thus, instead of 'have you read any of the following
and that the respondents were expressing genuine preferences and were not magazines within the last seven days?', perhaps try 'have you had time to read
merely reacting to such loaded words as 'routine jobs', 'independence', 'security', any magazines at all within the past seven days?' The latter creates a more
or 'manual job'. permissive situation for the non-reader. Second, brief interviewers to impress
Much inadvertent leading comes from a failure to state alternatives. The repeatedly on the respondents that accuracy is the prime requirement and that a
possible alternatives are merely implied or assumed to be so obvious as not to negative response is just as good as a positive response.
require stating; for instance, 'do you prefer being examined by a doctor of your As we have noted, classification questions such as occupation, income,
own sex?' instead of 'would you rather be examined by a male or by a female education and even age lend themselves very readily to prestige bias, and we must
doctor, or doesn't it matter which?' However, sometimes a question which is be particularly on our guard with these. However, not all cases of overclaim are
leading because it fails to state alternatives becomes less so when the alternatives due to prestige factors. In some cases, especially where there is a long list of
are given or read aloud to the respondent as multiple choices; for example, 'how possible answers, respondents are too eager to please; they want to give the
much trouble have teeth and gums been to you throughout life?' (This is leading, interviewer 'something' to put down so as not to disappoint her. Nor is
in that it assumes that the respondent's teeth or gums have been troublesome overclaiming the only likely source of error; in respect of the consumption of
and fails to suggest a negative alternative or to use a neutral filter question.) It certain goods, such as alcoholor tobacco, we often find persistent understatement.
is followed by (I)'A lot of trouble.' (2) 'Some trouble.' (3)'Little trouble.' (4) 'No People are reluctant to admit lack of knowledge. In straightforward knowl-
trouble.' One cannot help feeling that the wording of the question still suggests edge questions we generally get a higher percentage of 'don't know' responses
to the respondents that they must have had some dental troubles, but if the four than in opinion questions dealing with the same issues. This is because people
alternative answers were shown or read to them, they might well see that a are reluctant to admit that they do not really understand the question or are
negative answer was acceptable, which would reduce the leading effects of the unfamiliar with its contents. We must always keep this possibility in mind (if
question. As we have seen earlier, any multiple-choice answers offered to the possible, sift out those people for whom the question is meaningless) and not
respondent become part of the question and should be reported as such. assume that 'everyone' will know and will have an opinion on the subject of our
Leading bias often creeps in when we are formulating probes and follow-up inquiry. Similarly, if we start questions with phrases such as 'doctors say
or 'why1-questions; for example, 'how often do you go to your local super- that. . .', 'as you know. . .', 'many people nowadays . . .', 'research has
market?' followed by 'why don't you go more often?' -which suggests that the shown . . .' or with the mention of names of well-known personalities, we are
respondent ought to go more frequently. If the formulation of probes is left inviting prestige bias.
entirely to the interviewer, serious biases may result that will never be brought People do not like to think of themselves as fools, and it is best to avoid
to light. For instance, a question about voting intent might be followed by the questions that require the admission of foolish or reprehensible behaviour. Not
instruction, 'probe for importance of education policy, housing development, many mothers would, for instance, reply affirmatively to the question: 'Are
and defence programme.' It is almost impossible to imagine how one could there any foods or drinks that your children have that you know are bad for
phrase, on the spur of the moment, probes and follow-up questions that would their health?' In a survey dealing with oranges, greengrocers were asked
allow the respondents to indicate awareness of these issues and their bearing on whether customers often requested their advice on which brand to buy. This
voting intentions without at the same time leading them. Even if we phrase a was followed by the question, 'and do they mostly act on your advice?' Not
neutral 'whyf-question it may become a source of bias if different interviewers surprisingly, 98 per cent said 'yes.' If the question does contain some low-
read it out differently. In the well-known example: 'Why do you say that?' (see prestige alternatives, we should offer respondents a respectable 'out' - an
Payne, 1951) the stress can be placed on any of the five words it contains with opportunity to say that they intended to read the prestigous book mentioned
very different effects! (but had not actually done so) or that they had been too busy to become
acquainted with certain political issues.
SOCIALDESIRABILITY BIAS It has sometimes been found that the mode of questioning has an effect on the
We should always be aware that many factual questions are loaded with degree of social desirability bias: face-to-face methods such as interviews are
prestige. Some people will claim that they read more than they do, bathe more more likely to provoke respondents to put themselves in a better light than
often than is strictly true and fill more pipes from an ounce of tobacco than anonymous self-completion or postal questionnaires -but this is not always the
140 Questionnaire Design, Interviewing and Atfitvde Measurement Question wording 141
case. Similarly, telephone interviews may, or may not, be less prone to this kind not alone in engaging in this behaviour or in having these problems, and so these
of bias. Pilot work will help to overcome this problem, for example with the aid matters become more admissible. For instance, few mothers would admit that
of the split-ballot technique (see page 61 and table 8.1). one of their sons was a thief, but such admissions become easier when they are
offered the following alternatives: (I)'Never takes anything that belongs to
someone else'; (2) 'Has helped himself to someone else's things at least once or
QUESTIONS
ON SENSITIVE TOPICS
twice (including taking things belonging to other members of the family): (3)
It is not always obvious whether or not a topic is 'sensitive' to our respondents 'Has stolen things on several occasions.' Altogether, the creation of a permissive
or to a particular sub-sample of respondents, and so we must define 'sensitivity' atmosphere in the questionnaire as a whole, together with a guarantee of
in subjective terms: a topic is sensitive if some respondents find it embarrassing anonymity, can do much to overcome this problem.
and show reluctance to answer questions about it. A question can be
embarrassing for several reasons: because it asks about sociallv 'taboo' t a p i ~ QUESTIONNAIRES IN SCHOOLS (see also Chapter 7)
(child abuse in the course of religious
- rites); because it asks for information about
socially disapproved behaviour (extra-marital sex); because it could lead to self- Both open and closed questions are also widely used in research in schools.
-
incriminating answers (drug-taking); because it deals with matters which the Children tend to write much more slowly than they can read and they are often
taught to start every answer by repeating the question. Thus: Q 'Why do
respondent regards as~highl~perso~al-o_r_rExiVat_e_(inheritance); or because it may
put the respondent in a_+.b_all_.L$ii (questions about dropping litter). Social policemen wear a uniform?' A 'Policemen wear a uniform because people would
investigators sometimes forget how sensitive some people can be about certain not know they were policemen if they didn't.' From the researcher's point of
topics. view this is a time-wasting practice and so we would be well advised, when using
Such issues are really a special case of the social desirability bias discussed open-ended questions in schools, to print both the question and the start of the
above, but they may require more drastic measures, since it seems unlikely that answer so that the child merely has to finish the sentence: 'They wear a uniform
a nondirective question or a set of forced choices will suffice to obtain honest and because . . .'
complete answers. It is also worth mentioning that some of the embarrassment Altogether, survey work with school children requires additional care in
may be in the interviewers rather than in the respondents. There is the risk that piloting and fieldwork, and a different emphasis on some aspects. For example,
some interviewers, fearing a pained or hostile reaction from the respondent, will interviewing children in a school presents not only the additional problems of
-
find themselves sounding remorseful and making excuses instead of asking a
plainly worded question. Pilot work will often show that respondents are
perfectly-willing to answer stralgGGGardly phrased guestions about
such as incontinence, a criminal record, abortions, s m u- ~
--.- -_____
___---topics
E -i E t_- _-
and-run
-_
overcoming the children's possible fear of strangers and getting some genuine
and valid responses; but also there are the logistics of lessons and break-times to
cope with, and after the third or fourth interview every child in the school will
have heard a (probably inaccurate) version of the questions being asked! Group-
accidents, drug taking, and excessive alcohol consumption~~once tKey Z e administered questionnaires can overcome some of these problems, at least with
convinced that the information is relevant, that it will be treated ---
confidentially
, -.r=.--- children above the age of nine or ten in Britain. Even then, very great care is
(and ~referablvanonvmouslv) and that the interviewer is n o n - j u d @ E i ~ ~ ~ - required in the pilot stages, and it is best to have several fieldworkers present in
' In ;his context there is a pf&e, in the hands of the skilled investigator, for the each classroom to help with spelling, to answer questions and to distribute and
deliberately leading question such as: 'How old were you when you first started collect questionnaires. Bearing these precautions in mind, and with a promise of
to masturbate?' This puts the onus of denial on the respondents, the intention confidentiality, excellent results can be obtained quite rapidly from large
is to make them feel that this behaviour is quite common and well known and numbers of school children with the use of pre-coded questionnaires and many
can be discussed in a matter-of-fact way. Great care, however, has to be taken other closed techniques.
in interpreting the results.
Another way of overcoming denial tendencies is to hand the respondent a
INSTRUCTIONS
(SEE ALSO CHAPTER 6)
show-card on which the responses to a sensitive question have been given
numbers. All the respondent has to do is to call out the numbers of the applicable Every questionnaire will contain instructions to the respondent, while interview
items or responses; this is considered to be less embarrassing than putting such schedules will in addition contain instructions to the interviewer.
'admissions' into words. Belson (1975) has developed a very speciahed On self-administered questionnaires, instructions to the respondent will
technique whereby young delinquents can report self-incriminating informa- usually include answering procedures ('Tick oneJ;'Put a circle or ring around the
tion with the aid of a set of cards and a ballot box, so that the interviewer has no number of the item you prefer: 'Underline all that apply to youT)and routing
knowledge of the responses until after the respondent has left. directions or skip-to instructions (I'f NO, go on to question 26'; 'If you do not have
In milder instances of socially disapproved or private issues, merely offering access to a car, please turn to page 3'). Sometimes such instructions are more
the respondents a number of prepared categories can be sufficient. If such complex; for instance, respondents may be asked to choose the three most
matters are printed on an official-looking form, respondents realize that they are important items from a list and then place these three in order of importance by
142 Questionnaire Design, Interviewing and Attitude Measurement Question wording 143
numbering them 1 (for most important), 2, 3 (for least important). Or an should be briefed on ways of overcoming these. Particular attention should be
answering key may be provided (2 = very important; 1= fairly important; O = not paid to the wording of probes and to methods of verbatim recording.
important at all) which is to be applied to a whole list of items or questions. Due In addition to a review of definitions and procedural instructions, the briefing
attention must be given to layout, so that the respondent is never in doubt as to of interviewers will generally cover specific matters such as the selection of
how or where a response should be put, how many responses are required, and respondents, number of callbacks, the use of ratings, the distribution of product
does not have to zig-zag across the page to find a response box. Serious loss of samples, the handling of show-cards, details of recording techniques and cross-
data can result from ambiguous or inadequate answering instructions which checking procedures, as well as more general instructions about rapport,
lead respondents to tick several boxes when only one is required, or cross out probing, introducing the purpose of the survey and so on. Many research
responses which should be ringed or underlined. All too often questionnaires are organizations train their own interviewers and have standard handbooks that
sent out with too few answering instructions so that the returns are difficult to cover all routine interviewer problems and procedures.
process. When computer-assisted interviews are conducted, instructions, definitions
Self-completion questionnaires often contain introductory paragraphs. These and routing directives will usually appear on the screen in the particular code
help to set the scene and guide the respondent towards the answering pro- used by the system. Much the same applies to telephone interviewing.
cedures. For example:
into two or more equivalent subsamples and give each of them a slightly and aspirations. Altogether, we do not often find a one-to-one relationship
different form of wording. (The true answer is usually taken to be the average between attitudes or opinions, and behaviour.
of the subsamples.) It is probably true to say that internal checks would not Generally, in the case of factual questions we have the special difficulty that
detect a respondent who was really determined to mislead. we are not really trying to measure something 'inside' the respondent, but
In some surveys, it is possible to re-interview respondents, for example in rather that we are using the respondent as an informant, observer or reporter
panel studies, but we must be careful not to leave too much time between the about some past event, current status or possession, or future behaviour. The
original question and its repetition, or true changes in behaviour may occur in respondent is used as a repository of 'facts', an intermediary between the
the meantime. researcher and the required information. Our skills in question wording can
To ascertain the validity of factual questions, a variety of techniques is only get us half-way, at best: from the researcher to the respondent and back.
employed, usually known as external checks, where a second, independent The second half -from the respondent to the required information and back -
source of information is required, though not often available. Sometimes is entirely up to the respondent's ability and willingness to retrieve. When we
records can be used to check on certain points of information, for instance bear in mind that much of the required information, such as a routine
hospital records, factory-absenteeism figures, criminal records. Sometimes, the purchasing decision, is of low salience to the respondent and is unlikely to be well
total survey results can be compared with census figures, retail audits or remembered, then it seems almost inevitable that major problems of factual
manufacturers' production records, though these may themselves be open to validity will remain.
doubt. Occasionally, we may be tempted to use two informants to ascertain the
same facts or events, for instance husband and wife (with regard to marital RELIABKITY
AND VALIDITY OF ATTITUDE QUESTIONS
relations or joint financial transactions), parent and teacher (with regard to a
child's behaviour). However, there are many reasons why two observers or two Since attitudinal questions are more sensitive than factual questions to changes
participants in the same action may yet report differently on it, so that in cases in wording, context, emphasis and so on, it becomes almost impossible to assess
of disagreement one cannot say who is right and whether or not the criterion is reliability by 'asking the same question in another form'. It will no longer be the
more valid. This applies particularly to different observers reporting on the same same question. For this reason, we should not rely on single questions when we
individual. come to measure those attitudes that are most important to our study; we
Occasionally, some of the respondents to a mail inquiry are interviewed on should have sets of questions or attitude scales (see Chapters 9 and 11). Sets of
the assumption that the interview is more valid (see page 56). In other studies questions are more reliable than single opinion items; they give more consistent
quality checks are made; this means that some of the respondents who have results, mainly because vagaries of question wording will probably apply only to
been interviewed in the usual way are re-interviewed by a highly trained group articular items, and thus any bias may cancel out, whereas the underlying
of senior interviewers. This is especially worthwhile where a large number of attitude will be common to all the items in a set or scale. The reliability of a scale
temporary interviewers has been engaged. The second interviewer is usually can be assessed in the usual way by a correlation coefficient (see page 159)
allowed more freedom and can often track down a discrepancy, for instance to without having to ask the same questions twice.
a simple mistake in recording. In certain instances - in a hospital or The assumption underlying these procedures is that there is such a thing as
supermarket - it is possible to check on the responses with the aid of a 'true' attitude, which is also relatively stable, just as in the case of factual
observations. questions there are 'true' facts or events. However, since an attitude is more
We have so far considered mainly the type of questions that deal with facts or complex than, say, a respondent's method of travelling to work, it is unlikely that
behaviour in the past or present. Our test of validity is to compare the a single question will reflect it adequately. Also, the chances are that too much
respondent's account with what 'actually happened' - assuming that we can will depend on the actual question form and wording, on context, emphasis and
verify that by some other means. However, some surveys, such as election polls, mood of the moment, so that the results will be a compound of the (relatively
are intended to be predictive, and in that case the criterion of validity may be a stable) attitude and of these other (momentary) determinants -hence the poor
specified event in the future. This illustrates that a measure can have more than reliability of the single-attitude question. By using SETS of questions, provided
one purpose and thus more than one validity. A pre-election poll may be valid in they all relate to the same attitude, we maximize the more stable components
the descriptive sense if it gives us a true picture of the present political allegiances while reducing the instability due to particular items, emphasis, mood changes
of the respondents, yet it may show poor validity as a predictor of the popular and so on. None of the previously mentioned internal checks can readily be
vote three weeks later. The vote may be affected by any number of variables applied to opinion or attitude questions, but split ballots are sometimes used.
such as a press campaign, the weather, a political scandal or a domestic It may be helpful to introduce a distinction at this point between relative and
disagreement,.and therefore a valid description of the present may not be a good absolute consistency. If we need to know what percentage of our sample
predictor of the future. People are often poor predictors of their own behaviour, habitually travels to work by car, then this will be an absolute figure, give or take
so that statements of intent often lack validity when compared with subsequent a margin of error. If we ask the question more than once, we can reasonably
events, though they may well have been valid as statements of hopes, wishes hope to get a similar and consistent result. However, if we wish to know what
148 Questionnaire Design, Interviewing and Attitude Measurement Question wording 149
percentage of our sample has a positive attitude to saving, a great deal will such very different sets of results, or how we can predict contextual
depend on how the question is phrased. At what point, on a continuum from effects on a question, or in what ways we can ensure that respondents will all use
unfavourable through neutral to favourable, do we begin to include a the same frame of reference in answering an attitude question. We lack strong
respondent in the proportion of those with a positive attitude to saving? What theories about attitudinal constructs in people's minds, theories that might help
kinds of saving do we ask about? How strong is the respondent's attitude? us to understand what happens when we use language to trigger or activate
Suppose we have developed four different techniques (say, a free-response such constructs in order to reach and measure them. Without such theories, all
question about the best ways to take care of the future, a checklist, an attitude our tradecraft leaves us floundering, so that the problem of attitudinal validity
scale and a projective device using pictures), and that after due consideration of remains one of the most difficult in social research and one to which an adequate
the results we apply a cut-off point to each technique, with the result that we get solution is not yet in sight.
four different estimates of the proportion of respondents with a positive
attitude to saving. Do we now say that the questions are unreliable? Not
necessarily. We have to state, of course, that the answer will depend on how one Selected readings
defines and measures positive attitude to saving, but we may find that all four
methods intercorrelate quite well, and that group differences between subsam- Schuman, Howard and Presser, Stanley, 1981, Questions and Answers in Attitude Surveys,
ples are consistent. Thus, we may never be able to say what proportion of our Academic Press, New York.
sample has a positive attitude to saving in absolute terms, but we may well be in Belson, William A., 1981 The Design and Understanding of Survey Questions, Gower,
a position to show that the relative differences, such as those between middle Farnborough, Hampshire.
class and working class, or between older and younger respondents, are in the A detailed study of question wording and sources of misunderstandings.
same direction, irrespective of the technique used, and consistently show that Sudman, Seymour and Bradburn, Norman M., 1983, Asking Questions, Jossey-Bass, San
older people, or middle-class people, have a more positive attitude to saving. This Francisco.
type of consistency is well worth stressing, especially if it applies to techniques
Converse, Jean M. and Presser, Stanley, 1988, Survey Questions, Sage, London.
that apparently tap different 'levels' (see Chapter 12).
Some excellent practical advice from two experienced survey researchers.
The chief difficulty in assessing the validity of attitude questions is the lack of
criteria. What we need are groups of people with known attitude characteristics Belson, William A., 1975, Juvenile Theft: the Causal Factors, Harper and Row, London.
(criterion groups), so that we can see whether or not our questions can Payne, S.L., .1951, The Art of Asking Questions, Princeton University Press, Princeton.
discriminate among them. Occasionally, such groups can be found, for example An entertaining and still highly relevant book on question wording, with many
the adherents of a particular political party or the members of a particular examples.
church, but unfortunately people join groups for many different motives, and
there is no reason to suppose that group membership as a reflection of an inner
attitude is any more valid than a set of attitude questions. At best, there will only OCCUPATIONAL
GRADING
be a rough correspondence between the two indicators. We cannot necessarily Goldthorpe, J.H. and Hope, K., 1974, Social Grading of Occupations, Oxford University
predict behaviour from attitudes; nor are attitudes readily inferred from Press, London.
behaviour with any validity, nor is behaviour necessarily a more valid expression Examples and commentary on the status classification of occupations.
of an attitude than a verbal statement. The links between attitudes and
behaviour are complex, and group membership may or may not be a valid
criterion of a particular attitude. These issues are taken further in Chapter 9. PROBLEMS
OF RECALL
Belson, William A., 1962, Studies in Readership, Business Publications Ltd, London.
Robbins, Lillian C., 1963, 'The accuracy of ~arentalrecall of aspects of child development
and of child-rearing practices1,Journal of Abnormal and Social Psychology, LXVI, 261-70.
Inadequate question wording and questionnaire construction are not the only
causes of poor attitudinal validity. Since so much error and bias arises from
irregularities in procedures, changes in wording, faulty recording and failure to
maintain rapport we must do all we can in the spheres of interviewer selection,
training, briefing and supervision.
However, we also have to admit to a number of theoretical inadequacies.
When we sometimes despair about the use of language as a tool for measuring
or at least uncovering awareness, attitudes, percepts and belief systems, it is
mainly because we do not yet know why questions that look so similar actually
Some basic measurement theory 151
look at the linear scaling model in its classical form and then see how this model
has been applied to the field of opinion and attitude measurement, and to the
measurement of concepts, awareness and other subjective variables.
MEASUREMENT THEORY
The linear scaling model
WHATIS A MEASURE?
Let us start with a simple example. Suppose that we wish to know the length of
a table; we may take a metal foot-rule, hold it against the longer edge of the table,
and read off the answer: say, thirty-eight inches. This measure has, in principle,
Non-f actual questions two components: the 'true' measure and a second component which we may call
'error'. We shall, for the moment, ignore the kinds of errors-of-observation and
As we have seen towards the end of the preceding chapter, there are serious notation that may arise because we have been a little careless or imprecise, and
objections to the use of single questions to measure such non-factual topics as draw attention to the fact that we have used a metal foot-rule. Since we know
awareness, percepts, social representations, brand images, opinions, beliefs, that metal expands or contracts under different temperature conditions, part of
attitudes, values and stereotypes. Such issues are usually more complex than our answer is not a measure of length but a measure of temperature. Since, at
questions of fact; they have to do with states of mind, rather than with this point, we are not interested in temperature, we regard this aspect of the
behaviour or with events in the outside world, and are therefore difficult to measure as 'error' - in the sense that it is irrelevant to what we need.
measure and to validate; they are generally multi-faceted and have to be In this diagrammatic form, any measure could be represented as shown in
approached from several directions; above all, single questions dealing with such Figure 9.1.
sensitive topics are much more open to bias and unreliability due to wording,
question format and contextual effects. It is not unusual to find that responses
by the same group of subjects to two virtually identical items differ by as much
as 20 per cent or more, and we have no guiding principles by which to predict
how respondents will react to various forms of wording of what is basically the
same question.
Opinion researchers who commonly rely on single attitudinal questions (say,
whether or not the prime minister is doing 'a good job') are taking considerable
risks. However, they would argue that these risks can be minimized by using the
same question in successive surveys, so that an analysis of trends-over-time
becomes possible - whatever the question 'means' to respondents. They might Figure 9.1.
also~rguethat, since they have used this question many times in other surveys,
they know a good deal about its properties and correlates with other variables. The oval shape represents the reading or observation; the true measure is that
Also, something depends on what we want to know: perhaps all we need is a part of the reading which we want - it is an indicator of an attribute of
quick popularity snapshot of the prime minister, rather than an in-depth something or someone, for example length, musical ability, blood pressure; error
analysis of what people mean by 'a good job' and how this percept has been is that part of the measure which we don? want. The wavy line between the two
influenced by recent events. parts indicates that we are rarely certain about the exact size or location of these
But to the researcher who is hoping to rely on single questions about people's components. The diagram is an abstraction; it shows that we know our measure
attitudes to their doctor, to animal rights, to immigrants, to Green policies or to is 'impure' or 'contaminated' but not always by how much or in what way.
a certain brand of washing-up liquid, the unreliability of such single questions
will pose a serious problem. As we have indicated, if these variables are at all
important to the study, then the way forward may well be through the multiple-
EARLYSCALES IN THE NATURAL SCIENCES
question or scaling approach. To explain what is meant by scaling, we shall first You might argue at this point that the degree of contraction or expansion in a
have to consider some aspects of basic measurement theory; after that we shall metal foot-rule under normal everyday conditions is very slight and that
consequently the amount of error introduced in our measure is negligible - or
that we should have used a wooden ruler in the first place! The issue which this
152 Questionnaire Design, Interuiewing and Attitude Measurement Some basic measurement theory 153
example illustrates is, however, a general one which is present in every kind of of water, or the temperature of the average healthy adult human body.
science. Fahrenheit actually started his scale at the lowest point he could reach by using
Let us, bv wav of another familiar example, look at the ordinary household or
8
. d
a mixture of ice, water and sal ammoniac or salt - this must have seemed like
medical thermometer. Once again, the measure which we seek is an abstraction: 'absolute zero'at the time. Celsius originally proposed that the steaming point of
an indicator or an attribute of something or someone. 'Temperature', as such, is water should be zero and the freezing point loo0!Reaumur used a spirit
an abstraction, a scientific concept invented by humans; we try to measure it by thermometer, not a mercury one; instead of taking two chosen points and
observing the rise or fall of a column of mercury in a glass tube. We cannot say di:fiding the distance between them into 100 equal intervals, he considered the
that the position of the mercury is the temperature; the mercury is only rate of expansion of the original volume of spirit as the temperature rose. He
mercury. We are interested in the position of the mercury in the tube because defined loas an expansion of one-thousandth of the original spirit volume; this
we have made certain arrangements which will enable us to use it as an indicator led to a scale of eighty degrees or intervals between the freezing and boiling
of the state of someone (for example, a patient) or something (for example, the points of water. Today, in the physics laboratory, these early primitive methods
air in a greenhouse). These 'arrangements' are important. Some of them have of thermometry have been left far behind, but they serve to illustrate the basic
been designed to minimize the kind of error-component which we discussed problems of measurement that all sciences have to overcome.
above. For example, if there were air in the tube above the mercury, then this We can ask, for example: is it right to think of changes in temperature as lying
would make it harder for the mercury to rise, the more so as the mercury nears 'in a straight line'? This is the question of linearity, and the answer will lie in a
the top of the tube, and so we would have been measuring both temperature and better scientific understanding of the molecular processes that produce changes
- to an increasing degree - air pressure. This error would, in other words, in temperature in different materials. We can also ask: is it possible to measure
have taken the form of a systematic or correlated bias because the error gets 'pure temperature' and nothing else, not even the expansion of the glass? This
greater as the readings increase, and vice versa. The error is therefore is the question of uni-dimensionality and is closely linked to the earlier notions of
confounded, rather than uncorrelated or random. error. We know now that water boils at different temperatures at different
By drawing the air out of the tube above the mercury we usually manage to heights, in accordance with barometric pressures. This, and many similar
eliminate this error to a large extent but not altogether. We note further that the problems, such as the expansion rate of mercury or of certain gases, raises the
mercury moves in a tube made of glass; as the temperature rises, the glass problem of standardization: how can we be sure that any two or more
expands, the tube widens and the mercury rises more slowly - another thermometers will show the same readings under the same conditions? Then
'confounded' error. We trust and assume that the walls of the tube are smooth there is the question of consistency: can we rely on our thermometer to perform
and straight on the inside, for if they were uneven this would introduce a over and over again in the same way, yielding reliable readings which we can
random error, and if they were, say, closer together at the top of the tube than trust?
at the bottom, we would have another 'confounded' error - though one that Another complex issue is the question of the units of measurement. As we have
works in the opposite direction - yielding an over-estimate rather than an seen, three different inventors produced degrees of different sizes, which may
under-estimate. cause us perhaps nothing worse than some irritating conversion problems, but
Another important arrangement with thermometers is that they have a scale. we may ask whether at certain points -say, when something melts, or freezes,
In the eighteenth century, when the thermometer was being invented and or burns, or dies - are there not also qualitative, as well as quantitative,
improved by men such as Celsius, Reaumur and Fahrenheit, it seemed a good differences in temperature? Or, to put it another way, are the units really
idea to think of temperature as a straight line, a linear dimension, and to divide interchangeable all along the scale? Are, say, the five degrees from o0 to 5OC
this line into equal intervals (grouped according to the decimal system). The idea (which cover thawing) the 'same' as the five degrees from 95' to 10o0C (which
of using equal units along a continuum probably came from measures of cover the boiling of water at sea-level pressure)? And what about the beginning
distance such as miles or leagues, or perhaps from ells or yards or from measures and the end, the top and bottom points? Do all measures require a zero-point? A
of weight such as pounds or kilograms. However, two difficulties arose: one was maximum? Can they have negative readings? Last but not least, how can we be
the question of the size of the unit or interval, the other was the problem of sure that there is a precise and consistent relationship between the abstract
finding a suitable starting point. After all, with distance, or length, or weight, a quality we call 'temperature' and the readings we obtain from mercury moving
zero- or nil-point at the outset seems obvious - but can one think of any point in a capillary tube? This is the problem of validity to which we shall have repeated
as 'nil temperature', as no temperature at all? There was also the desire to invent occasions to return.
something that could be used in a practical way both in the laboratory and in To sum up then: one answer to the question 'what is a measure?' is to describe
everyday life, covering the range of temperatures that were of interest at that the properties of a scale. A scale must satisfy the conditions of linearity and uni-
time - the idea of measuring the temperature of the sun must have seemed dimensionality; it must be reliable (consistent); it must have units of measure-
quite out of the question! And so each of the inventors of the thermometer ment (not necessarily equal ones); these units will, if possible, be interchange-
chose a zero-point and a top-of-the-scale point that seemed convenient for range able; the scale needs some 'anchoring points' obtained through standardization
and were thought to be'anchored in Nature', be it the freezing and boiling points or in relation to fixed, observable events, to give meaning to the scores; and most
154 Questionnaire Design, Interviewing and Attitude Measuremed Some basic measurement theory 155
particularly, the scale must be valid, in the sense that it measures the attribute so successful in the physical sciences still appropriate when we have to deal with
it sets out to measure. Perhaps surprisingly, the simple yardstick did satisfy all behavioural and social phenomena? This remains a very controversial question
these requirements, once it was standardized (such a standard can still be viewed on which the last word will not be spoken for a long time to come.
on the outside wall of the Greenwich observatory). But if the linear-scaling approach makes demands which are technically hard
The basic notions of measurement are not hard to grasp, but they become to satisfy, are there any practicable alternatives? There are many positive
more difficult to achieve as we move from the natural sciences, through the answers to this question, but all of them entail losses as well as gains. One type
biological and medical sciences, to the behavioural and social sciences. of loss has to do with the richness of human experience. When we describe a
rainbow in terms of a spectrum of lightwave frequencies, or motherlove in
terms of a score on an attitude scale, we become acutely aware that something
THEPROPERTIES OF A SCALE OF INTELLIGENCE
important is lost. On the other hand, the richer, more subjective approaches -
In the light of these requirements, let us look at a test of intelligence. Will it for example, historical descriptions, diaries, literary accounts, free-style inter-
satisfy the conditions of linearity and uni-dimensionality? Some complex views, school reports, group discussions - are less reliable, less valid, less
statistical techniques will have to be used to establish this for the particular scale objective and less comparable. In the strict sense of the term they are not data
in question, for we are no longer dealing with a simple foot-rule. (Even if uni- or measures at all. They are rather like statements about hot or cold made by
dimensionality can be established, how happy are we to think of a complex set people before the thermometer was invented: subjective, imprecise, not always
of qualities, such as intelligence, in terms of a straight line?) Will it be reliable, in consistent and with little hope of consensus or comparability. The movement
the sense that repeated administration of the test will yield virtually the same towards making things more 'objective' and 'scientific' is an attempt to get away
results? Leaving practical problems aside, the answer will probably be'Yes', since from subjectivity, to make measures more 'true', more reliable, more precise,
we have fairly accurate techniques for assessing the degree of reliability. Will it more replicable and more comparable even if this does entail certain losses. But
have units of measurement, and, if so, will they be equal and interchangeable? it has to be said that the present 'state of the art' in measurement in the human
Here we are on more difficult ground. The test is likely to yield a score in terms sciences is often so crude and so poorly developed that the losses may well
of points, which are treated as if they are units of equal interval which are outweigh the gains. It thus becomes a key responsibility of competent research
interchangeable in many cases, but this remains a questionable issue. workers to choose the data-generation and -collection techniques which are
Can the scores be given meaning, by relating the individual's score to some most appropriate and most likely to 'do justice' to their problem and their
anchoring points andlor standardized ranges? To this, the answer will be 'Yes' subject-matter, a bitter and difficult choice between too much over-
- any test will be published together with its norms for various age groups, simplification for the sake of 'good measurement' and a subtler, more flexible
ability groups etc., and frequently the test scores will be converted into an approach which may not yield valid, replicable results. We can only hope that
Intelligent Quotient which offers, not a zero-point but a mid-point, the 'average' increases in the diversity and sophistication of approaches to measurement will
of 100 IQ. It is, however, highly debatable whether IQ points or intervals are eventually release us from this dilemma.
equal units and whether they are interchangeable, though they are frequently
treated as such.
Finally, will the test be valid? Will it really measure 'intelligence'? The first and Other types of measures
crude answer to this will be 'Yes' because the test will have been correlated with
some earlier, well-established IQ tests. But this merely moves the problem a
stage further back, for how do we know whether those tests, the 'criterion ORDINAL
SCALES
measures', are valid? At this point we might have to enter two complex sets of Let us now look at some well-understood departures from the linear scaling
arguments, one having to do with techniques of validation and one with the model. We have previously seen that, ideally, such a scale should consist of
nature of intelligence. We may note, therefore, that even such a well-tried-out identical, interchangeable units. If at certain points along the continuum we
technique as a modern test of intelligence may - from the measurement point need greater precision or accuracy, we can sub-divide such units and obtain
of view - have some rather dubious attributes. finer-grained readings, but this does not alter the basic principle. However, what
if the units are known to be unequal or cannot be shown to be equal? In such
cases we look to see if they have a sequence, or an ordinal property - in other
ARESCALES APPROPRIATE TO OUR DOMAIN?
words, whether they can be ranked. In everyday life we quite often use a ranking
Of course, it is also possible, indeed desirable, to turn the argument on its head. approach: children may be ranked from top to bottom in a class; athletes, or
Instead of asking whether tests of intelligence satisfy the criteria of linear horses, may be ranked in the order in which they finish a race; the pop charts
scaling, we might well ask whether the linear-scaling approach is appropriate to may rank music recordings in order of their popularity. The important point to
the complex set of qualities we call'intelligence', or can ever hope to do it justice? note is that an ordinal or ranking scale tells us nothing about the intervals
More broadly, to what extent are the methods of measurement which have been between its points: the winning horse may have been ahead by no more than a
156 Questionnaire Design, Interviewing and Attitude Measurement Some basic measurement theory 157
neck, whereas the third one could have been trailing by several lengths. So long Statistical treatment
as we are certain about the ordinal properties of the scale, we can make use of
these for the purposes of measurement. There are statistical techniques that will Any accumulation of measures will require some form of management and
enable us to do this, and ordinal scales - while necessarily less precise than interpretation, even if this is done in a purely intuitive way. More commonly, the
equal-interval ones - should otherwise have most of the other properties of data are subjected to statistical analysis, aided by computers. Typically, the
linear scales. measures for each respondent will be entered into a computer (or, in the case of
A special sub-variant of ranking scales are scales using the paired comparisons small-scale manual analysis, on to an analysis sheet laid out in the form of a grid),
method. This can be useful when we have a small number of objects (say, a and thereafter they will be manipulated in a variety of ways. We may calculate
dozen) and we wish to see how or in what order they should be arranged. For various averages, we may compute percentages, we could examine the data for
example, we might have a set of pictures, animals, smells or holiday resorts to statistical significance, we could work out correlations and we might go on to
place in order of preference, if we can. Of course, we might ask each of our various forms of multi-variate analysis such as multiple regression or factor
respondents to rank the stimulus objects, but a more general method would be analysis. This should enable us to 'make sense of the data'; that is, to test our
to arrange the objects in all possible pairs and present these to each respondent hypotheses, to compare results for various sub-groups, and so on.
in random order, pair by pair. Instead of having to arrange a dozen pictures, or It is very important to realise that the statistical treatment of linear interval
holiday resorts, in order of preference (which may be very difficult), all that the scales is quite different from the treatment of nominal or categorical measures.
respondent has to do is to choose between two members of each pair. If there is The properties of the linear scale permit us to treat the scores as integers which
some kind of underlying dimension or continuum on which all the objects can be may be added, subtracted, divided, multiplied and so on. This implies that they
placed, this will emerge from the statistical analysis, but even so, it dyield not can be analysed by means of statistical techniques applicable to interval-type
an interval scale, but a rank-order scale as above. This method offers advantages scales. These techniques are quite powerful; they include the familiar 'average'
where the dimension is uncertain and ranking would be difficult for the entire or mean, the variance and the standard deviation, analysis of variance, many
set of objects, but with human respondents it can only be used with relatively types of correlation coefficient and most multi-variate methods. The results can
small sets of stimuli - even twelve objects yield sixty-six paired comparisons to be tested for statistical significance (to check whether the observed findings
make, and the numbers very soon rise to unmanageable magnitudes. could have arisen by chance) by means of t-tests, F-tests, Z-tests and others,
which many computer programs provide as a matter of course.
Not so with nominal data. Since they lack interval properties and have no
NOMINAL
MEASURES underlying continuum, they cannot be added, multiplied or squared. In the initial
We now come to the measurement of nominal or categorical data. These are stages, after entering our data, all we can do is count. For each of our categories
data which have no underlying continuum, no units or intervals which have (of eye colour, party affiliation or whatever), we shall wish to see how many of our
equal or ordinal properties, and consequently cannot be scaled. Examples might respondents endorsed it: how many people in our sample had blue eyes, how
be: eye colour; the daily newspaper you read; the political party you last voted many had brown, how many had grey, and so on. No numerical value can be
for; the type of novel you like to read; ownership of consumer durables, such as attached to these counts or entries; all we can do is count the frequency of their
a car, a home freezer, a motor mower; the brand of cigarette you smoke; your occurrence. After that, we might wish to compare sub-samples: do men have
country of birth; and so on. There is no underlying linear scale; instead, there are grey eyes more often than women? Are older respondents more prone to vote
a number of discrete categories into which the responses can be classified or for party P than younger ones? To make allowances for differences in sub-
'coded', but the categories can be placed in any order and they have no numerical sample sizes, we shall wish to convert these frequency distributions into
value, nor is there assumed to be an underlying continuum. Essentially, the only percentages, and then study the differences between the groups - but it is not
possible score for each category is a binary one: yeslno, or presentlabsent. Thus, possible, say, to calculate averages or standard deviations.
if my eyes are brown and I am given a list of eye colours, then my answers will Distributions of this kind have to be analysed by means of non-parametric
be 'no1, 'no', 'no', until I come to 'brown' when I shall give a 'yes' response, and methods, which are fewer and less powerful than the interval-scale techniques
then 'no1, 'no', 'no' again to the remainder of the list. There is no order to the already mentioned. Chief among them is the Chi-squared test (see Appendix 11),
categories since no colour is 'better' or 'stronger' or 'bigger' than any other. which allows us to compare our observations with chance expectation, or with
expectations based on specific hypotheses. (Warning: the Chi-squared test must
never be applied to percentages but only to the raw frequencies in our tables.)
There will obviously be problems when we try to correlate two nominal
Beyond this, and in a sense departing still further from the linear-scaling model, measures with each other, or with a scaled variable, though there are some
are numerous other techniques using multiple items; for example, perceptual measures of association that can be used, and there are some non-parametric
techniques such as the semantic differential; sociomefry; various grid techniques; and techniques for the multi-variate analysis of nominal or ordinal data. However, in
projective techniques. These will be discussed in subsequent chapters. general the statistical management of nominal measures has fewer and less
158 Questionnaire Design, Interviewing and Attitude Measurement Some basic measurement theory 159
powerful techniques at its disposal than are available for the management of human body. New, more accurate and more valid measures only began to
scaled data. develop when scientists acquired a better understanding of the underlying
Between the scaled and the nominal measures lie the ordinal or ranked processes that determine temperature change.
measures. These data do not have interval values but are ranked or ordered on
an underlying continuum, such as a pupil's position in class or the results of a
race, ranked from frst to last. There is a special set of statistical techniques The reliability and validity of scaled measures
which make use of the ordinal properties of such data. The data do not have
interval properties but neither are they completely discrete categories, and this Up to this point we have been chiefly concerned with two important attributes
has led to the development of measures of rank correlation, tests of significance of scales: dimensionality (the presence of a single, linear, underlying continuum)
for ranked data and so forth. However, these techniques are less powerful and and equality of intervals and their statistical treatment. We now turn to a
relatively limited in number compared to those available for scaled or metric consideration of reliability and validity, two properties which constitute the
data. essence of measurement or data generation of any kind. Each of them is
This may explain why researchers frequently 'bend the rules' in order to be important and, as we have seen previously, they are related to each other.
able to use parametric techniques of analysis. Take, for example, a five-point In the preceding chapter we dealt with reliability and validity in very general
classification of socioeconomic status (SES). Strictly speaking, we have five terms, and also more specifically in respect of questions dealing with facts and
classes or categories in no particular order, with no numerical scale-interval behaviour, and in respect of single questions dealing with attitudes and beliefs.
values, and no underlying scale. It follows that we would have to confine We must now deal more specificallywith the reliability and validity of scales and
ourselves to the use of Chi-squared for significance testing, and we would not other multi-question approaches.
be able to use ordinary (product moment) correlation coefficients to relate SES
to scaled data such as income, birthweight or holiday expenditure. However,
suppose we made the assumption that socioeconomic status is a linear
dimension, from low to high in a straight line; and suppose, further, that we We shall start with reliability, for adequate reliability is a precondition to validity.
made the assumption that each status category is not independent but has a Reliability means consistency. We need to be sure that the measuring
rank, from first to fifth. In that case, we would have an ordinal scale, which instrument will behave in a fashion which is consistent with itself; that a very
would make our analysis easier and more powerful. Better still, suppose we high proportion of the score on every occasion is due to the underlying scale
made the assumption that the status categories are not merely ranked but have variable, with a minimum of error. If we find differences between readings on
interval properties; that would be really helpful. Now we can calculate status the same instrument on two separate occasions, or when applied to two
averages for different occupational groups, we can use correlations to link SES different objects or respondents, we must then be sure that these are 'genuine'
with numerous scaled values such as educational achievement tests, and we can differences or changes in the subject of measurement, and not differences which
include SES in major multi-variate analyses to show its importance in could be attributed to inconsistenciesin the measuring instrument or to changes
'explaining', say, the outcomes of different forms of psychotherapy. in the attendant conditions. The notion of reliability thus includes both the
In strict measurement terms this is wrong, and it follows that the parametric characteristics of the instrument and the conditions under which it is
techniques for statistical analysis are not applicable. Yet researchers frequently administered - both must be consistent. In terms of Figure 9.1 (page 151), it is
take liberties with the requirements and assumptions on which these statistical the error component, the impurity, which produces the inconsistencies and
techniques are based in the hope that the techniques will prove sufficiently unreliability which we shall seek to reduce. For instance, when comparing lung
robust to withstand a certain amount of abuse, and that the ensuing results and X-rays of a patient before and after treatment, the doctor must feel confident
conclusions will not be too misleading. that the observed differences are real differences in the patientjs lungs and not
One last word about early thermometry, where practitioners seem to have differences due to imperfections in the emulsion, inconsistent exposure times or
struggled with difficulties not unlike our own. It is obvious now that the original postural changes of the patient.
thermometers were rather like pseudo-scales. The units were arbitrary and had Reliability, or self-consistency, is never perfect; it is always a matter of degree.
different sizes and different meanings according to different measurement We express it in the form of a correlation coefficient and, in the social and
systems; the assumption of the linear expansion rate of mercury and of certain behavioural sciences, it is rare to find reliabilities much above .90. The square of
gases was not demonstrated until later; there was either no zero-point or there a correlation coefficient expresses the percentage of shared true variance; thus,
were arbitrary zero-points located to suit human convenience; and for a long a reliability coefficient of .90 means that the two measures have .81 or 81 per
time the readings were unreliable because of technical imperfections in the cent in common - they overlap, or share a common variance, by just over four-
technique of glass-blowing. Also, it took a great deal of international compara- fifths. If the reliability of a scale or other measure drops below .80 this means
tive work and organization before universal standards were fixed for the that repeated administrations will cover less than 64 per cent of the same
melting points of various metals or for the average temperature of the healthy ground, and that the error component is more than one-third; such a measure
160 QuesfionnaireDesign, Inferviewing and AffifudeMeasuremenf Some basic measurement theory 161
will come in for serious criticism and might well have to be discarded or rebuilt. want to draw from it -in this instance, the concurrent validity of changes in the
Reliability may be measured in several different ways: (i) by repeatedly lungs and the predictive validity of estimated recovery rate, prognosis and so on
administering the scale to the same sample within a short period (tesf-retest (which may be different).We also note that what we want to measure, and what
reliability). However, this may produce resistance, as well as a practice effect - the instrument is 'supposed to measure', may often be an abstract attribute of
in a sense, it will no longer mean that the 'same' test is being administered under some kind (for example 'response to treatment', 'likely outcome') for which, at
the 'same' conditions. To avoid these problems we can use (ii) the internal the time, we may have no other, 'true' criterion measure.
consistency method, usually associated with Cronbach's Alpha coefficient and its Inevitably, these problems are magnified in the social and behavioural
variants; (iii) the split-half method; and (iv) the parallel-form method. All these sciences.An achievement test in arithmetic may possibly give us a valid measure
methods yield reliability measures in the form of correlation coefficients. The of this child's ability to solve fhis set of problems foday.In this limited sense it could
parallel form method applies to the situation where the test-building procedure be said to be 'valid' because it has concurrent validity, but usually we ask such
has yielded two equivalent forms of the test, identical in every way except for the tests to bear a heavier burden. For example, we may want to treat this set of
contents of the items. Having an alternative form of a measuring instrument arithmetical problems as representative of all such problems at this level and to
can be very useful in a before-and-after design, so that respondents will not predict that this child could solve anofher set of similar problems tomorrow; that
receive the same measure twice within a short time period. The split-half is, we may wish to generalize from this score. How valid is such a generalization
method meets the latter problem in a different way: the group of items likely to be? Or we may wish to use this test to assess the success of a new
comprising the measure is divided into two halves at random, and the two halves teaching method which is intended to benefit the less-able pupils; would the test
are then intercorrelated. be a valid measure by which to assess the efficacy of the new method? And
The internal-consistency method rests firmly on classical scaling theory. If the might it also be able to show which pupils would benefit most: does it have
scale is expected to measure a single underlying continuum, then the items predictive validity?
should have strong relationships both with that continuum and with each other. But how do we know that the test is a test of arithmetic? That it measures
While we cannot observe the former, a scale will be internally consistent if the 'what it is supposed to measure', an abstraction? A common-sense answer
items correlate highly with each other -in which case they are also more hkely would be to examine the contents of the items. This is known, somewhat
to measure the same homogenous variable. Items are more likely to satisfy these disparagingly, as 'face validity' and holds many dangers. For example, out of the
requirements if they are reliable, that is if they have low error-components. entire domain of arithmetic, how have these items been selected? What do they
Since coefficient Alpha gives us an estimate of the proportion of the total represent? For what level of achievement? How well balanced are the items in
variance that is not due to error, this represents the reliability of the scale. terms of content? How pure are the items? In addition to knowing some
arithmetic, the child must know how to read and write, how to concentrate and
solve problems, how to handle language and so on, and these abilities may, in
turn, be related to social background factors. Thus, part of the score will show
Validity may also be expressed as a correlation coefficient, but this has a different the child's level of achievement in arithmetic, and another part will measure
meaning from that of the reliability coefficient. There are other ways of several other skills, which will not merely introduce some random error into the
expressing validity, depending on the type of validation being used. In principle, data but a correlated bias, so children from certain backgrounds will tend to have
validity indicates the degree to which an instrument measures what it is higher scores. We have to satisfy ourselves that these dangers have somehow
supposed or intended to measure, but this is a somewhat vague statement, been minimized by the way in which the test has been constructed.
especially since what we usually seek to measure is an abstraction, so that One method of establishing concurrent validity might be to correlate the
measurement must take place indirectly. Thus, using our previous example of scores with an external criterion - with some other, 'truer' measure - but
the doctor looking at two X-ray plates, and assuming that the negatives are which one? Teachers' ratings or examination results are often used in such
reliable and are indeed of the same patient before and after treatment, we note contexts but these tend to be unreliable, and anyway, why are they 'better'?
that the doctor is not really interested in the differences between two Good criterion measures are notoriously hard to find.
photographic negatives as such, but in changes in the patient's lungs. If the There are other ways of approachingthe problem of validity. Sometimes we
technique is valid, then there will be close correspondence (concurrent validity) can use consfrucfvalidify. There are times when we have good theoretical grounds
between the X-rays and the changes in the lungs. From these, the doctor will try for making certain predictions (for example, that children who score higher on
to draw conclusions about further treatment, prognosis and so forth (predictive an IQ test will also do better at arithmetic), and so we regard the fulfilment of
validity). If the technique had poor validity, this might lead the doctor to draw a set of such predictions as evidence of the test's validity. Certainly we should be
erroneous conclusions. Note, however, that at best the technique can only rather suspicious of an arithmetic test on which the brighter children did less
reveal changes in the patient's lungs - the rest is inference. Thus we note that well (always assuming that the intelligence test was reliable and valid in the first
each measurement technique can have more than one validity (indeed, place). Finally, we can examine the test's predicfiue validify; for example, high
potentially an infinite number of validities), depending on the conclusions we scorers should be able to give change correctly if they are put behind the counter
162 Questionnaire Design, Interviewing and Attitude Measurement Some basic measurement theoy 163
of a shop, they should get higher marks in an arithmetic examination at the end reliable. This is particularly the case in the field of attitude scaling (see below and
of the year, and they might become more successful bookmakers or actuaries! Chapter 11).
An aptitude test for motor mechanics might have high validity for predicting
examination results at the end of a training course, but may have poor validity
in predicting success as a garage owner, and no-one has yet succeeded in Scores and norms
designing a test, or a battery of tests, that would predict success in final
university examinations. The usefulness of predictive validity, if attained, can be By itself, a numerical score on a measure will be almost meaningless. To state
very great - but we may have to wait a long time to achieve it, and even then that someone has scored thirty-two on an ethnic prejudice scale will tell us very
we must make sure that the criterion measure is sufficiently valid and reliable little, unless we can place such a score in its context. Once a scale has been
for our purposes. developed we therefore seek to give it a scoring system and to obtain norms for
To sum up, a measure may be used for a number of distinct purposes, and it it. For example, in the case of the test of arithmetical achievement which we
is likely to have a different validity coefficient for each of them. There are also mentioned before, we could administer the test to a large number of school
different types of validity: children - preferably to representative samples of the relevant age groups -
and obtain frequency distributions, percentile curves, means and standard
deviations and so on for each school year, and perhaps separately for boys and
(i) content validity, which seeks to establish that the items or questions are a well- girls or some other sub-groups of concern. We might also seek to correlate our
balanced sample of the content domain to be measured ('face validity' is not test with other tests to give more 'meaning' to the scores. Once we have done
really good enough); all this, we can state that a score of, say, forty-nine achieved by a ten-year-old
(ii) concurrent validity, which shows how well the test correlates with other, well- boy is very much below average for children of his age, or that his arithmetic
validated measures of the same topic, administered at about the same time; achievement is well below what his IQ test would suggest, so that some remedial
(iii) predictive validity, which shows how well the test can forecast some future work might be needed in order to correct this 'under-achievement'.
criterion such as job performance, recovery from illness or future examina- In the more complex and multi-faceted domain of attitude measurement we
tion attainment; do not generally take such a 'positivistJ approach, but it remains necessary to
(iv) construct validity, which shows how well the test links up with a set of place our scale scores in context, by showing the statistical results of giving the
theoretical assumptions about an abstract construct such as intelligence, scale to substantial groups (preferably probability samples) of the population of
conservatism or neuroticism. concern.
As we have seen earlier, validity and reliability are related to each other.
Above all, reliability is a necessary (though not sufficient) condition for validity: Improving the reliability of attitude scales
a measure which is unreliable cannot attain an adequate degree of validity - its
error component is too great. On the other hand, a measure may be highly The traditional method of measuring attitudes is by means of attitude statements.
reliable and yet invalid. Consider, for example, three radio direction finders who In the next chapter we shall see the ways in which these may be composed and
are homing in on the transmissions from a plane, a ship, or a fox that has been assembled into an item pool, but for the moment we shall treat them as any
radio-tagged. By triangulating their observations and plotting the results on a other items that form the raw material of scale building. The purpose of such a
map, they will obtain a small triangle (or, in the ideal case, a point) from within scale would be to place each respondent on an attitudinal dimension or
which the transmissions must have originated. The smaller the triangle, the continuum, but at the start of our research we may not know whether such a
more reliable the measurement - but by the time the triangle is drawn, the dimension exists, whether it is likely to be linear, or whether it might perhaps
location of the plane, ship or fox is ldsely to have changed, perhaps by thirty to break up into several different dimensions.
fifty miles, and so the measure will no longer be a valid indication of position. We Though in writing the statements or questions we shall obviously be aiming
can hope to improve on this situation by taking repeated observations, plotting at content validity, we must be acutely aware that each item may contain a
course and speed and thereby inferring a valid location (though with a fox substantial error-component which will make it unreliable as a measure, and
carrying a radio collar, this might not work!). Thus we see that, if we can that there is no way of 'purifying' single items. Thus, suppose we were building
establish good reliability then we are, as it were, 'in business: we may not yet some parental attitude scales and that we were working on items for a rejection
have much evidence of validity, but there is hope that with further work the scale, a way of finding out whether a parent shows a tendency to reject his or her
scale's validity may become established at an acceptable level. children. This is obviously a subtle problem, but we might try a permissively
In general, the more difficult it becomes to find a suitable external criterion, a phrased item such as: 'If we could afford it, we would like to send our children
strong construct system, or some other method of validation, the more to boarding school'. But how can we know whether this item really belongs in
researchers will concentrate on content validity and on making their measures a rejection scale, for instance whether a mother who responds to this item with
Some basic measurement theory 165
164 Questionnaire Design, Intemiewing and Attitude Measuremenf
'agreeJis actually inclined to reject her children and wants to get them out of the
way? For all we know, in her eyes attendance at some famous boarding school
opens up a glittering future for her children and so, though she loves them
dearly, she is willing to make sacrifices to give them a better future. Conversely,
a 'disagree' answer may not mean greater love for the children, it might merely
indicate a certain political orientation. If we refer again to Figure 9.1, we can say
that in items of this kind the error component is likely to be particularly large and
variable, so that a single item will be highly unreliable.
The commonly adopted solution to this problem, though by no means the
only one, is to try to apply the linear-scaling model. In principle, the argument
can be presented as shown in Figure 9.2.
If we take not one but a set of items all designed to measure the same
underlying attribute or continuum in a variety of different ways, and if we score
these items and add up the scores for each respondent, then this total score will
be a better, more reliable measure because all the M-components of the various
items will be added together, whereas the E-components will, the argument
suggests, be non-additive and random. In so far as we can accept this argument,
it suggests that the more items we use, the higher will be the reliability of the
total score. There is, however, a point of diminishing returns (which is probably
just as well, or we should be using tests with hundreds of items!), so that most
scales tend to consist of no more than a couple of dozen items each.
u
Figure 9.3.
Figure 9.2.
166 Questionnaire Design, interviewing and A t t i h k Measurement Some basic measurement theory 167 '
The argument above is clearly open to question. A lot depends on the interpersonal skills. Do all these items have something in common which could
construction and weeding-out of the items. There are also serious problems of be called 'secretarial ability'? And could we refine the item pool so as to yield a
overlap between the items. If there is too much overlap of the M-components of smaller number of questions which would enable us to give each student a
the items, then there will be redundancy and one-sidedness in the total score. numerical score on that ability? O r again, suppose we want to grade groups of
The content area has been 'tapped' but in a limited way, and the underlying recruits to the armed services on 'physical fitness' by giving them a number of
dimension will be somewhat lopsided - rather like an intelligence test based tests: for endurance, for eyesight, for lung volume, heart rate, doing press-ups,
principally on space-form and memory items, for example. Even more serious is weight lifting and so on. How can we find out whether all these different test
the possibility of substantial overlap between the E-components of the items. In procedures have something in common which may be called 'physical fitness'
that case, far from being random, the errors will reinforce each other and create and on which new recruits could be graded? Factor analysis may provide some
a bias. Suppose that in building an intelligence test we have included quite a few answers to these questions.
items which, although they approach intelligence in different ways, all require Because of the prodigious amount of calculation involved, the use of a
the ability to read rather quickly in order to understand each item, or need some computer is essential, as a rule. We start by creating a correlation matrix. This is a
familiarity with coloured blocks making abstract patterns. In such cases children square table with all the variables listed both across the top and down the side;
who were fast readers, or who had played a lot with coloured blocks, would have each cell in the body of the table contains a correlation coefficient between two
an 'unfair' advantage and the total score would contain an important bias. variables, so that the matrix as a whole contains every possible correlation
In Figure 9.3 we have presented the same problem in a somewhat different between any variable and every other variable. For example, a 20 x 20 matrix
way. This drawing suggests that the pool of items or statements should be would contain 380 correlation coefficients in two identical triangular sets of 190
'balance& that is, it should try to do justice to the multiple facets of an attitude coefficients each.
domain. For instance, in Chapter 4 we gave the example of measuring attitudes Before going further, it is worth giving this matrix some careful inspection. It
towards the police. The depth interviews and preliminary conceptual frame- would be interesting, for example, if we found a group of variables which
showed high intercorrelations, for these might have underlying dimension in
work might well contain sub-areas to do with uniforms, with law and order,
common. O r we might find a large number of very low correlation coefficients.
with crossing the road, with domestic crime, with police programmes on
television, with motoring, with feelings towards authority and so on. The Does this mean that our data have a 'poor structure', that the variables mostly
drawing suggests that items dealing with every one of these aspects, and more, do not 'hang together' very well? And what if we find some negative
might all have something in common, namely an abstraction which we would correlations? Is this merely a matter of the way some variables have been scored,
call Attitude to the Police, as well as other components each dealing more or do we really have some variables which go down when most other variables
particularly with burglaries, speeding, or dislike of a particular television go up? If so, do such variables belong in this matrix at all?
programme, as well as error components. Now we must start the factor analysis itself. We need a software package
Next, we would need to conduct an item analysis: the procedure of statistically which contains a factor-analysis program (or a variant of factor analysis called a
'weeding out' items that do not 'belong' or that have too much error variance. principal-components analysis), which we shall apply to the correlation matrix.
This would help to 'purify' the scale, and to establish its reliability through The program will take care of the calculations, and of certain transformations
internal consistency and, as we shall see, there are a variety of methods for which have to precede the analysis, and will produce a set of output tables. The
achieving this. Ultimately, we might apply a correlational procedure such as first output table will usually show how much 'structure' there is in the
factor analysis or principal component analysis (see below) to purify the scale still underlying dimensions by giving a measure of the size of each factor. If there is
further and to suggest the underlying dimension. These procedures would also a strong, simple, underlying structure showing that all or most of the variables
enable us to produce a more precise scoring method. essentially measure the same dimension (secretarial ability, say, or physical
fitness), then the first output table will show a list of one very large percentage
value followed by a list of much smaller ones. More typically, the first table will
Factor analysis show one fairly large percentage, two or three medium-sized ones and a residual
list of smaller percentages which will be ignored. This may be interpreted to
The procedure of factor analysis is an analytic statistical tool which may enable mean that the underlying structure is not uni-dimensional but two-dimensional
us to find out what (if any) are the chief underlying dimensions of a set of or three-dimensional. Thus, it might show that physical fitness is not a single
variables, attributes, responses or observations. Suppose, for example, that we concept or attribute but requires more than one measure. Sometimes the first
want to develop a measure or test that could be used in a secretarial college to output table will show no large percentages at all, merely a set of small ones. This
measure 'secretarial ability' at the end of a training course. We might assemble would be interpreted to mean that essentially there is no common dimension. In
a pool of questions, some dealing with keyboard skills, some with office that case the variables mentioned earlier do not really belong together at all, they
management, some with inter-office communications such as telephones or fax should not be added together, and it is not possible to give each subject a single
machines, some with minor account-keeping and spread sheets, and some with .score or grade.
168 Questionnaire Design, Interuiewing and Attitude Measurement Some basic measurement theory 169
The factor-analysis program will have applied a cut-off point, whereby the ( Figure 9.4.
remainder of the analysis will be concerned only with underlying dimensions Here are some possible reasons for drinking. How important is each one of
which are substantial. The second output table is likely to show a list of all the these to YOU as a reason for drinking?
tests, measures or attributes in our pool. Next to each of these there will be one
or more figures that look like correlation coefficients. These lists will be headed 1
'Factor 1', 'Factor 11' etc. If, for example, the analysis has suggested a two-factor (For each question place a tick in the appropriate column)
structure, then this table will show two lists of coefficients (called 'factor
loadingsJ).Essentially, this is a transformation; what the program has done is to Not at all Slightly Fairly Very
important important important important
transform most of the variance in our correlation matrix into something simpler
and more manageable: a two-factor structure.
This factor structure is, however, not the final stage. The factors produced by 1 1. I drink when I am feeling tense.
this first transformation may be somewhat arbitrary, they may lack meaning I
and may be difficult to interpret. They are not unlike, say, a grid on a map or a
globe. We may draw the zero meridian through Greenwich, or through New I
I
I
2. I drink to be sociable
I I I
Delhi; either way, it will be an arbitrary convention which does not represent
any land of geographical or geological feature or process, and in that sense it is
not 'meaningful'. Also, whichever way we choose to draw our grid, this will I I I I I I
make no difference to the distances between cities or the location of mountains 3. I drink when I am feeling miserable
or other features. If we were to move or rotate the zero meridian to run through I
New Delhi, this would merely mean that the geographical co-ordinates of any
city or mountain would have to be altered. The features themselves would not I I I I I
change, nor would their relative positions. 4. I drink because the people I know drink.
Various procedures have been developed to help us to identify and interpret
the underlying dimensions we may find. These will require the factor-analysis
I I J
program to engage in 'rotation': the program will make a number of attempts I I
iI
I I
We now come to the interpretation of the results of the factor analysis. As we
9. I drink to celebrate social occasions.
have already noted, the first part of the interpretation concerns the overall
underlying structure. Is there a single dimension, or are there several, or none?
In the second part, we shall seek to give meaning to each factor, as far as we can,
and perhaps give it a label. We do this by careful scrutiny of the factor loadings.
For example, we might find that the list of Factor I loadings show high loadings
/ lo. I drixk when I just can't cope with tdngs I I
for key-board secretarial skills, and low loadings for social or interactive skills,
while the Factor I1 loadings show the opposite pattern. This might suggest that
secretarial skills are not all on a single, unitary dimension, but are composed of
170 Questionnaire Design, Interviewing and Attitude Measurement Some basic measurement theory 171
two sets of variables on two separate dimensions: key-board related and social/ VARIMAX ROTATED FACTOR MATRIX
interactive, and we might label our factors accordingly. AFTER ROTATION WITH KAISER NORMALIZATION
Often, however, we find more complex results in a factor analysis. For FACTOR 1 FACTOR 2
example, there may be one strong, major factor which enters into every variable,
and then several more 'specific' factors which are each represented by a cluster
of factor loadings. Thus, 'physical fitness' might be one such general factor,
followed by a handful of measures that have loadings only on Factor I1and which
might, on examination, turn out to be tests requiring muscle strength. Another
set of loadings might define Factor 111-these might have to do with endurance
and with recovery from effort; and there might be a fourth factor with loadings
derived mainly from sensory performance tests such as visual and auditory
acuity.
Various factor analyses can yield different patterns, and the interpretation of
the results is a matter of experience and judgement with which a computer
cannot help. Sometimes, however, it is possible to apply a different form of Figure 9.6.
rotation, called 'obliquefrotation. Up to now we have assumed that every factor
produced by the analysis was independent of every other factor. In, say, a two- dimensional structure underlying our variables, sometimes with revealing or
factor result a person could score high or low on the first factor, but this would surprising results. Thus, if we have specific hypotheses about our domain of
enquiry, for example that one score can be used as a summed measure of physical
not tell us whether that person was likely to score high, or low or somewhere in
between on the second factor. The two factors were independent and unrelated fitness, then we can use factor analysis to test such a hypothesis for construct
to each other. That is why they were drawn at right angles to each other in a validity. At other times, perhaps in a new area of research, factor analysis can
graphical representation. However, it sometimes happens that this is not the help us to chart new structures, which may lead to new measures and new
theories.
most meaningful way of representing our data. In that case, we might try an
oblique rotation - a solution which assumes that the factors are, or can be, to
some extent related to each other.
Factor analysis has a number of uses. As we shall see, the procedure is often Figure 9.4 (on page 169) is an inventory of ten items describing motives for
employed in scaling and in the development of other kinds of measures. If a drinking alcohol. The factor-analysis output shows in Figure 9.5 that 51.7 per
respondent is to be given a score on an instrument, then we must make sure that cent of the variance is accounted for by the first factor, 17.4 per cent by the
the scale or inventory is uni-dimensional. Otherwise we must either shorten the second factor and that the remaining percentages are negligible. Thus we have
scale by using only those items that have high loadings on the factor we require, essentially a two-factor structure. Figure 9.6 shows the rotated factor loadings
or we must give each respondent two or more scores, depending on the factor for the ten items. To interpret the results, we note that items 1,3,5,8, and 10
solution. We can also use the factor loadings for each test or item to give weights have substantial loadings on Factor I; these are all items which express internal
to the responses, thus producing refinements in the scoring process. motives for drinking. The remaining items have high loadings on Factor I1 and
Perhaps the greatest value of the technique is in showing us the nature of the mostly express external motives for drinking. The diagram in Figure 9.7 shows a
two-dimensional factor plot, with the ten items clustered around the two axes.
FACTOR EIGENVALUE PCT OF VAR CUM PCT
Recapitulation
The classical linear-scaling model requires that a measure shall have four
important attributes:
(I) linearity and uni-dimensionality;
(2) at least adequate reliability;
(3) at least adequate validity;
(4) a scoring system, and some statistical norms.
In this chapter we have tried to show how each of these objectives can be
Figure 9.5. achieved by traditional means.
172 Questionnaire Design, Interviewing and Attitude Measuremenf Some basic measurement the0 y 173
Selected readings
Allen, Mary J., and Yen, Wendy M., 1979, Introduction to Measurement Theory, BrookslCole
publishing Company, Monterey, California.
A useful, non-mathematical introductory text.
Rossi, P.H., Wright, J.D. and Anderson, A.B. (eds), 1983, Handbook of Sumey Research,
Academic Press, New York.
See chapter 7.
DeVellis, Robert F., 1991, Scale Development: Theories and Applications, Applied Social
Research Methods Volume 27, Sage, London.
Torgerson, W.S., 1958, Theory and Methods of Scaling, Wiley, New York.
For advanced students.
Guilford, J.P., 1956, Psychometric Methods, McGraw-Hill, New York.
An older but still very useful text. See especially chapters on psychophysical
methods and on rating scales.
Bernstein, I.N., 1976, Validity Issues in Evaluative Research, Sage.
Cronbach, Lee J., 1990, Fifth Edition, Essentials of Psychological Testing, Harper and Row,
New York.
A classic basic text. See especially on validity and reliability.
Dunn, Graham, 1989, Design and Analysis of Reliability Studies, Oxford University Press.
For the advanced student.
Streiner, David L. and Norman, Geoffrey R., 1989, Health Measuremenf Scales, Oxford
University Press.
A very useful explanatory text aimed at medical researchers. Deals with reliability,
validity, response bias, telephone interviewing and many other relevant topics.
Carley, Michael, 1981, Social Measurement and Social Indicators, Allen & Unwin, London.
Figure 9.7.
FACTORANALYSIS
In the following two chapters we shall discuss the various classical scaling Gorsuch, R.L., 1974, Factor Analysis, Saunders, Philadelphia.
methods that have been developed to measure attitudes, and compare these to A useful basic text.
the linear scaling model. In subsequent chapters we shall deal with less
traditional forms of attitude measurement, and with various other multiple- Rummel, R.J., 1970, Applied Factor Analysis, Northwestern University Press, Evanston, Ill.
item techniques which diverge from the linear-scaling model. Harmon, H.H., 1967, Modern Factor Analysis, University of Chicago Press, Chicago.
Kim, Jaeon and Mueller, Charles W., 1979, Factor Analysis, Quantitative Applications in
the Social Sciences, No. 14, Sage, London.
usually dormant and are expressed in speech or behaviour only when the object
of the attitude is perceived. A person may have strong attitudes for or against
foreign travel, but these become aroused and expressed only when some issue
connected with travel abroad arises - or when confronted with an attitude
questionnaire! Attitudes are reinforced by beliefs (the cognitive component) and
often attract strong feelings (the emotional component) which may lead to
particular behavioural intents (the action tendency component).
Attitudes, like many other determinants of behaviour, are abstractions -
though they are real enough to the person who holds them. There is no limit to
the topics about which people may have attitudes - from war and peace, God
and eternity, to toy trains, roses, or frozen fish. It would be possible to argue
persuasively that in the final analysis everything in life depends on people's
attitudes. Since attitudes are so important in domains such as politics, marriage,
religion, food preferences, social change, education, fashion, child rearing, ethnic
No amount of statistical manipulation will produce a good attitude scale unless prejudice, communication and many others, it is not surprising that social
the ingredients are right. The most advanced scaling techniques and the most psychologists have devoted much time and ingenuity to finding ways of
error-free procedures will not produce an attitude scale worth having unless measuring them. We become particularly aware of the strength and pervasive-
there has first been a good deal of careful thought and exploration, many depth ness of attitudes when we try to change them - through the processes of
interviews and repeated conceptualizations, and unless the ingredients of the communication, advertising, education, conversion, propaganda, military drill,
scale - the attitude statements - have been written and rewritten with the alcohol consumption, brainwashing, and others.
necessary care. We shall therefore not approach the topic of attitude scaling at Most attitudes are given labels such as liberalism, pacifism, feminism, anti-
once, as some textbooks do, for in the past too much attention has been paid to Moldavianism, authoritarianism, strictness, vegetarianism, and others. Such
the technical and statistical problems of s c a h g and not enough to the labels should be regarded with caution, for it is of vital importance to know what
psychological problem of what is being scaled. This chapter is devoted to the the attitude measure is supposed to be about. This is more easily said than done,
writing and selection of attitude statements; the next chapter will deal with the for most attitudes are parts of a wider compound of values, beliefs and feelings
construction of attitude scales. and are themselves made up of several components or sub-areas. It is only too
An attitude statement is a single sentence that expresses a point of view, a easy to impose a preconceived framework (which may or may not be misguided)
belief, a preference, a judgement, an emotional feeling, a position for or against on an attitude, but this could result in unintentionally measuring the wrong
something. For example: 'The people in power know best'; 'The police can't be attitude, or only part of an attitude, or too many attitudes all at once.
trusted: 'Children bring a husband and wife closer together'. Attitude state- Nor do attitudes necessarily 'hang together' in a logical way. There are, for
ments are the raw material of attitude scales. They should be phrased so that example, no logical reasons why people who favour capital punishment should
respondents can agree or disagree with the statement. Usually, after the pilot also be against homosexuality and against socialized medicine, but in favour of
work, a large number of attitude statements (probably somewhere between restrictions on immigration, yet research has shown that these are all part of a
sixty and 100)is assembled to form an item pool. These items will be analysed and much wider orientation to many social issues.
submitted to a scaling procedure (see next chapter). If all goes well, the outcome Our thinking on the nature of attitudes has been rather primitive. Most of the
will be a much smaller selection of items (say, between one and two dozen) time we tend to perceive them as straight lines, running from positive, through
which together have the properties of an attitude scale that can yield a numerical neutral, to negative feelings about the object or issue in question. Our attempts
score for each respondent. The analysis will also have thrown light on the topic at measurement then concentrate on trying to place a person's attitude on the
or dimension which the scale is designed to measure. straight line or linear continuum in such a way that it can be described as mildly
positive, strongly negative, and so on -preferably in terms of a numerical score
or else by means of ranking. There is no proof, however, that this model of a
What is an attitude? linear continuum is necessarily correct, though it does make things easier for
measurement purposes. For all we know, attitudes may be shaped more like
The study of attitudes has a long and complex history in social psychology, and concentric circles or overlapping ellipses or three-dimensional cloud formations.
this is not the place to engage in elaborate theoretical exposition. Suffice it to say The model of the linear continuum or dimension is not always easy or
that, for the purposes of verbal measurement, most researchers seem to agree appropriate. For instance, the positive part and the negative part may or may not
that an attitude is a state of readiness, a tendency to respond in a certain manner be linear extensions of each other. In studying the attitudes of mothers to their
when confronted with certain stimuli. Most of an individual's attitudes are children we have found that rejection is not the exact opposite of acceptance (as
176 Questionnaire Design, Interviewing and Attitude Measurement Designing attitude statements 177
measured by two separate attitude scales); apparently, a variety of other
components enter into the attitude of a rejecting mother that are not present in opinions
a loving and accepting mother. Moreover, it is possible for some respondents to
be ambivalent and to score highly both on acceptance and on rejection. It would
clearly have been a mistake to assess these attitudes by means of a single
acceptance-rejection continuum. Again, the degree of differentiation at one end
of an attitude continuum may be very different from that at the other end.
Thus, if we considered how we felt about peace, we would find it easy to deal
with the negative end - statements about the benefits of war, courage and the attitudes
will to fight, and national unity - but at the positive end we would very soon
run short; there is not all that much that one can say about love for peace. Does
this mean that the units or intervals will be smaller, or more numerous, at one
end of the scale than at the other? One is tempted to divide the dimension into
units of some kind. Such units, while not strictly 'legitimate', may help us to
compare groups of people with one another, but they are not comparable from
one scale to another. values
Attitudes have many attributes. So far, we have talked mainly about their
content - what the attitude is about. An attitude, however, also has intensity. It
may be held with greater or lesser vehemence. To some people, the exploitation
of animals may be but of passing interest or concern, whereas to others it may
be of great importance and propel them to leading positions in the animal personali
liberation movement. We might expect the latter to agree or disagree more
strongly than the former would to a series of statements dealing with the
treatment of animals. This attribute of intensity can be very important in
understanding how attitudes function. Figure 10.1.
Some attitudes are more enduring than others. For instance, a man's political
beliefs may be fairly stable throughout his lifetime, whereas his attitudes to wrong in his opinion that Mexicans have a high crime rate; but his underlying
tennis or to gambling may undergo multiple changes. Similarly, some attitudes anti-Mexican attitude remains unaltered, and he will soon find some other belief
go much deeper than others and touch upon a person's fundamental philosophy with which to bolster his hostile attitude.
of life, while others are relatively superficial. Again, some attitudes seem to be Typically, attitudes do not exist in isolation within the individual. They
more embracing than others; they lie at the base of more limited or specific generally have links with components of other attitudes and with the deeper
attitudes and beliefs, thus predisposing individuals in a certain way toward new levels of value systems within the person. For instance, suppose we are having
attitudes and experiences that may come their way. For ease of understanding, dinner with a friend in a restaurant and suggest to him that he might like pork
social psychologists make a rough distinction between these different levels, chops, which are on the menu. Our friend declines. At this point, we might say
calling the most superficial one 'opinions', the next one 'attitudes', a deeper level to ourselves that we are dealing with an opinion, with a relatively superficial and
'values' or 'basic attitudes', and a still deeper level, 'personality'. These rather not immutable food preference of the moment. However, suppose we started
vague distinctionsbetween ddferent levels of attitudes must be thought of as more questioning our friend. We might find that he is a vegetarian, a Jew or a Muslim,
versus less enduring, deeper versus more superficial; relatively stable versus and that his refusal to eat pork chops has to do with a more general and deeper
relatively changeable; and more general versus more specific. Furthermore, underlying attitude to pigs and other animals as food. This in turn may be linked
these levels must not merely be thought of as the different layers of a cake; there to an underlying value system that has to do with the role of humans ois-d-ois
are relationships and patterns of connection between these layers, rather like other creatures, with religious proscriptions concerning 'clean' and 'unclean'
what is sometimes called 'the tree model' (see Figure 10.1). animals, or possibly with other taboos. An apparently simple matter of food
This diagram should not be taken too literally. Its chief function is to warn preference may in fact be a key to the individual's fundamental philosophy of life.
against treating opinions and attitudes too much as isolated units, and to So far, we have talked about relationships in terms of depth. However,
illustrate the problems we face in trying to bring about a change in anyone's attitudes are also related to each other 'horizontally', at the same level. Some of
attitudes. At the most specific level, that of opinions, change is relatively easy to the most interesting research contributions in social psychology have been
bring about so long as the underlying attitude is not involved. Thus, it may not concerned with such interrelationships. We find, for instance, that racial
be too difficult to convince a man with strong anti-Mexican views that he is prejudice against one minority group is usually associated with prejudice against
178 Questionnaire Design, lnteroiewing and Attitude Measurement Designing attitude statements 179
several other groups, and with the glorification of the individual's own group. nevertheless are very interested in them and are influenced by advertising
The word 'ethnocentrism' has been coined to reflect such as underlying claims concerning vitamin content. We begin to find various links; health-
predisposition, which in turn is part of a personality syndrome known as consciousness is one, of which vitamin-consciousness may be a part, though a
authoritarianism and which contains many other attitudes besides ethnocen- rather unruly one. Child care may be another link, associated especially with
trism. We are dealing here with the difficult problem of isolating one attitude women and with small babies. A general tendency towards self-medication,
from a considerable number of others to which it is linked and correlated and toward taking pills and tablets of all kinds, may represent yet another
which, in their turn, may also be part of underlying value systems and connection. There may also be a link with health foods and an interest in
personality syndromes. It is difficult to know where to draw the line. alternative medicine. And we could continue. This is a part of social psychology
Exploratory work suggests all kinds of interesting links and patterns with where clinical acumen, intuition and a capacity for 'listening with the third ear'
neighbouring and underlying fields, but we also need, for measuring purposes, are invaluable and where chairbound, preconceived frameworks may constitute
a scale covering some relatively well-demarcated area. a real hindrance.
Interrelations such as these follow no logic except the 'psycho-logic', the logic We must next decide, after doing perhaps thirty or forty preliminary
of feelings and emotions. It is most important to realize that attitudes are only interviews, what it is we wish to measure, and we will then be in a position to
very rarely the product of a balanced conclusion after a careful assembly of draw up a conceptual sketch of the attitude clusters in question, with their likely
evidence. As a rule, attitudes are acquired or modified by absorbing, or reacting linkages and possible undercurrents. We should also have some idea of the
to, the attitudes of others. We like to maintain the fiction of rationality and function that our scale is to play within the context of the research as a whole.
impartiality in reaching our conclusions, but, in fact, attitudinal predispositions This may tell us how general or specific, and how deep or superficial, our scale
play a very considerable part. Attitudes can be highly emotional, both in the should be and what sub-areas it should include or leave out. As a rule, the more
sense of irrational or illogical and in the sense of arousing powerful needs and aspects of a particular attitude can be included, the more likely it is that scores
ego defences. When we ask a highland Peruvian woman to boil her drinking will be obtained that will mean something in terms of that underlying attitude,
water to kill germs, we are, in her eyes, also asking her to adopt the white man's rather than in terms of one or two particular aspects of it. But it must be
magic, to betray her ancestors and to make a fool and a renegade of herself in her remembered that the same attitude may express itself in different ways in
village. To us this may not seem very reasonable at first, but this is where different people, while some people may have no such attitude at all.
research can help -by showing the underlying attitude links, with their strong Using a Sentence Completion technique (see Chapter 4) can also be helpful at
emotional connections. this stage.
(A) 'It is important for men to look masculine.' (Though this item may nowadays is that you cannot distinguish easily the male from the
seem meaningless, people do say such things in interviews and female.' (Too long, too clumsy, dangers of double negatives.
the statement seems likely to be effective.) Content seems appropriate; needs rewording.)
(B) 'I do not have any hesitation in trusting a woman doctor.' (This (R) 'The father should be the only one who spanks a child.' (Problem:
statement may 'work' very differently for male and for female will this get mixed up with attitudes to corporal punishment? It
respondents: the issue of same-sexldifferent-sex doctor may get assumes that a child will be spanked.)
mixed up with the purpose of the scale: here, to measure (S) 'A woman's place is in the home.' (This is a widely used item in
occupational sexism. Also, the word 'trusting' might have to be many scales, especially those measuring political conservatism.
replaced.) Should we avoid it?)
(C) 'There is a right and a wrong way of expressing emotions for a (T) 'No real man would want to help with housework.' (Probably a
man and for a woman.' (Clumsy wording.) useful statement; the phrase 'real man' nicely worded to
(D) 'Men who wear scented aftershave are disgusting.' (Wording may provoke.)
be too strong, may yield poor frequency distribution. Perhaps (U) 'A man who wears makeup must be a homosexual.' (Two
take out 'scented'? Does the statement tap the issue of equality problems: (1) will the homosexuality issue get mixed up with the
between the sexes, or is it measuring something else as well?) gender equality issue? (2) again, a general observation, which
(E) 'Public toilets should be made unisex.' tells us little about the respondent's own views - it is possible to
(F) 'I like to see men carrying babies.' (This might be misunderstood disagree with the statement, yet still to feel that men should not
to mean that men might, one day, be able to get pregnant. wear makeup. See statement P above.)
Otherwise, a good, personalized expression of view.)
(G) 'I should like to see more male secretaries.'
(H) 'Men should be the breadwinners.' (This looks like a good The above comments illustrate some of the issues that arise in selecting and
statement, but some exploration may be needed of the term editing attitude statements for an item pool. We need items that are clear,
'breadwinner'.) personalized expressions of feelings, containing not more than one issue. The
main topic addressed by the item must be part of the object of our scale, and the
(I) 'Men and women are different but for most things irrelevantly so.'
(Double-barrelled. What would a 'disagree' response mean?) items must be relevant-to the issue being measured. Descriptive observations
are best avoided. If a person agrees, say, that 'most old people live in the past', this
(J) 'I see no reason why a woman should not propose marriage to a
man, if she wants to.' (Could be shortened. May produce does not tell us whether the respondent likes or dislikes old people (unless we are
problems with double negatives - best rephrase it to get rid of prepared to make the assumption that living in the past is something of which
the 'no'.) the respondent disapproves, but such an assumption would be risky). After
(K) 'I like to see men pushing prams.' (A nice, colloquial phrasing.) reading each statement, try to 'translate' it into 'I likeldislike' whatever the
statement is about. If this is not clear, then the statement may be too vague or
(L) 'I would have less confidence in a woman professional than in a
male one.' (This is probably better than statement B above, but it too neutral, and it will be impossible to score the responses. Try to ensure that
is so general that it may prove unreliable: different respondents the item pool is reasonably balanced. In the item pool referred to in Workshop
may give very different meanings to the term 'professional'.) VI, there would have been perhaps a dozen sub-areas (gender equality at work;
(M) 'Boys who play with dolls should be firmly discouraged.' (Likely dress and makeup; and so on), each of which would have to contain at least half-
. ,
a-dozen statements, in roughly equal numbers of 'pro' and 'con' items. Thus, an
to 'work' very well.)
(N) 'Jobs like changing the baby's nappy could be done as well by the item stating that men are better drivers than women would have to be balanced,
father as by the mother.' (Clumsy wording. Also, some respond- if possible, by items stating that there are good and bad drivers among either sex,
ents may take it literally: the issue is not really how competently or that women are better drivers than men. At the end of the day, the item
it is done but about gender differences in the parenting role.) analysis and scaling procedures will sort out the good items from the bad ones
( 0 ) 'All-male clubslsocieties are a good thing.' (May need some and will show whether the whole area can be represented by a single, uni-
rephrasing, but should work well.) dimensional scale or will need several scales or sub-scales.
(P) 'Women drivers aren't as good as male drivers.' (This statement
One problem with attitude scales is that they have a limited life span.
may well be effective, but as a rule we should avoid statements in Sometimes the cause is obvious: the objects of concern have changed or
the form of general observations (for example 'dogs foul the disappeared. For instance, scales built to measure attitudes to the Vietnam War
pavement'), on the grounds that it tells us too little about the or to the Soviet Union can no longer be used in their original form. At other
respondent's own feelings.) times, the debate has moved on. For example, it seems likely that in most
(Q) 'One thing I do not like in the way young persons are dressed countries and in most periods there will be a political dimension of radicalism-
184 Questionnaire Design, Interviewing and Attitude Measuremenl Designing attitude statements 185
conservatism (perhaps called by different names, such as left-wing versus though the measures dealt with such complex topics as civil liberties, ethnic
right-wing, Labour versus Tory, Social Democrat versus Christian Democrat) equality, war and peace, political participation, equal rights for women and
in the electorate. However, the issues that define this dimension will change attitudes to local government. This study illustrates that, with sufficient care in
over time. For example, issues such as corporal punishment, abortion, homosex- the pilot work and much pre-testing in different countries, it is possible to
uality, women's suffrage or premarital sex probably no longer divide the voters develop attitude measures which meet the requirements of scaling, which are
in most Western countries in the way they once did; other issues have come to comparable across countries and age groups, and which deal with complex and
the fore, and the same dimension will have to be re-defined. Sometimes, too, subtle issues as understood by children and adolescents.
social mores and attitudes change, to make many of the scale items irrelevant.
Here, for example, are some items from a 1959 cluvalry scale (Nadler and
Morrow, 1959) developed in the United States: I WORKSHOP VII: UNSCALED ATTITUDE STATEMENTS
I
A gentleman should remove his hat and hold it in his right hand when a lady The following attitude statements appeared in a questionnaire that
enters the elevator in which he is a passenger. was sent out to a sample of married women. They were designed to
It is inexcusable when a man escorting a lady fails to help her on with her coat. assess women's attitudes to detergents, soap flakes and the polluting
effects of detergents. There were no other attitude statements.
No right-thinking person would question the sanctity of motherhood.
In accompanying a lady anywhere at night, a gentleman should never fail to 'Regular soap flakes cannot be used in a washing machine.'
offer his arm. 'All detergents are equally good.'
'Enzymes in detergents should be outlawed.'
The same considerations apply when we take a scale from one country to 'Soap flakes get clothes as white as detergents.'
another: we must not assume that, after translation, it will measure the same 'Detergents make clothes cleaner than soap flakes.'
'thingJ in both countries; we have to make sure that it does (usuallyby means of 'I would buy a more expensive soap if it was non-polluting.'
factor analysis). Translation can subtly alter the meanings and overtones of an 'If we continue to use detergents we will harm our lakes and rivers.'
attitude statement, or the same statement may have a changed significance in a
different social context. We could criticize these statements for their clumsy wording and
There are accessible collections of questionnaire items and attitude scales in poor grammar and for the fact that they seem to be trying to measure
several countries (see Selected Readings). These can be a useful input when we two or three quite different issues, but more important is the fact that
are building a new scale, but existing attitude scales should never be uncritically these statements do not form a scale. This means that each statement
applied in a different country or in a changed social context. stands on its own, like any other single attitudinal question, with all the
Some attitude scales are crude and obvious in their purpose, giving rise to the attendant risks of unreliability and poor validity. Such a set of
notion that scales cannot be subtle or sensitive and can only deal with the banal statements looks like a short scale, and it may impress the uninitiated,
and the obvious. This is far from true, as has been illustrated in Workshop VI but it will not yield a combined score. Each item could be analysed
where the items were designed to uncover an emergent attitude or set of separately (though, as we have seen, this would be most unwise), but
attitudes. that would defeat the purpose of using attitude statements. As they
It is even possible to develop complex attitude scales for use with school stand, these seven statements are misleading and of little use: they
children, provided extensive pilot work has ensured that the children are aware give the superficial impression of being 'scientific', but they do not
of the topics of concern, and that they do have opinions about them. (With constitute a measure. Unless they are to form part of a much larger
children, it is helpful to have answering categories such as 'I have not heard or item pool and will be subjected to item analysis and scaling proce-
thought about this' and 'I have no opinion', in addition to agreeldisagree.)Some dures, they should be abandoned.
years ago a cross-national study of political socialization and civic education
(Torney et al., 1975) was undertaken in ten countries, which included Finland,
Iran, Germany, Italy and the United States. Within each country, representative
samples of children were drawn from either two or three age groups (ten to
eleven-year-olds, fourteen to fifteen and seventeen to nineteen-year-olds) -
twenty-two sub-samples and over 30,000 children in all (some countries did not
collect data for some age levels). Despite differences in ages and languages, it
proved possible to develop attitude scales and other measures which were
shown to have near-equivalent factor structures in all twenty-two sub-samples,
. . .
186 Quesfionnaire Design, Intemiewing and Affitude Measuremenf
Selected readings I1
ATTITUDE
THEORIES AND MEASUREMENT ATTITUDE
SCALING
Fishbein, Martin (ed.),1967, Readings in Attitude Theory and Measurement, Wiley, New York.
This very useful collection contains many classical papers dealing both with
measurement and with theory.
Fishbein, Martin and Ajzen, Icek, 1975, Belief, Attitude, Intention and Behaviour, Addison-
Wesley, Reading, Massachusets.
A well-known general text on attitude theory and the links between attitudes,
intentions and actions.
Pratkanis, Anthony R., Breckler, Steven J. and Greenwald, Anthony G. (eds), 1989,
Attitude Structure and Function, Lawrence Erlbaum Associates, Hove, East Sussex.
Deals with more recent developments in attitude theories.
This chapter is devoted to an account of the classical methods of measuring
Adorno, T.W., Frenkel-Brunswik, Else, Levinson, D.J. and Sanford, R.N., 1950, The attitudes through the use of attitude scaling. Other methods of measuring
Authoritarian Personality, Harper, New York. attitudes will be discussed in the following chapters.
A classic study of prejudiced attitudes, showing the development of concepts and Attitude scales consist of from half a dozen to two dozen or more items -
scales from depth interviews as well as the links between different ethnic prejudices, usually attitude statements - with which the respondent is asked to agree or
ethnocentrism and authoritarianism.
disagree. In previous chapters we have noted the dangers that attach to the use
Nadler, R.B., and Morrow, W.R., 1959, 'Authoritarian attitudes toward women and their of single questions to measure attitudes, and we have discussed the generation
correlates1, J. Social Psychol., 49, 113-23. and composition of a pool of attitude statements, preparatory to the construc-
Source of the chivalry scale cited in the text. tion of an attitude scale. However, more important than the number of attitude
Torney, Judith V., Oppenheim, A.N. and Farnen, Russell F., 1975, Civic Education in Ten statements used is the fact that they have been scaled: they have been selected
Countries, Wiley. and put together from a much larger number of statements according to certain
statistical scaling procedures. Because of this, we must not judge the relatively
small number of attitude statements in a finished scale at their face value; they
WRITING
ATTITUDE STATEMENTS
are the outcome of a complicated process of item analysis and represent a
Edwards, Allen L., 1957, Techniques of Attitude Scale Construction, Appleton-Century-Crofts, synthesis of all the preliminary thought and work described in Chapters 4,s and
New York. 10.
Attitude scales are relatively overt measuring instruments designed to be
used in surveys, and we must not expect too much of them. They are not
designed to yield subtle insights in individual cases and should not be used as
clinical instruments. Their chief function is to divide people roughly into a
number of broad groups with respect to a particular attitude, and to allow us to
study the ways in which such an attitude relates to other variables in our survey.
They are techniques for placing people on a continuum in relation to each other,
in relative and not in absolute terms.
Principles of measurement
The traditional methods of attitude measurement are best viewed and evaluated
in relation to the linear-scaling model (see Chapter 9). This model has the
following requirements:
cohesive, they should'hang together' to measure the same dimension with as attitudes, then probably the Likert procedure will be the most relevant. If we
little extraneous variance as possible. wish to study attitude change, or the hierarchical structure of an attitude, then
2. Reliability - the indispensible attribute of consistency. Traditional scaling Guttman's method might be preferable. If we are studying group differences,
methods are often strong in this respect. then we shall probably elect to use the Thurstone procedures. Each type of scale
3. Validity -the degree to which the scale measures what it sets out to measure. does one thing well, and, if this is what our research needs, then this is the type
As we have seen, at present it is often impossible to find a sufficientlyreliable of scale we will want to use.
and valid external criterion against which to validate an attitude scale.
4. Linearity and equal or equal-appearing intervals - to make quantitative
scoring possible. Social-distance scales
To these requirements a fifth has been added later, namely: In 1925, Bogardus brought out a social-distance scale, subsequently revised, that
has been widely used in the United States and illustrates some of the problems
5. Reproducibility - when we say that a man weighs 150 pounds, we mean that in scale construction.
the pointer on the scales will move from zero to 150 but will cover none of
Direcfions: According to my first feeling-reactions, I would willingly admit
the remainder; in other words, the figure of 150 refers, not to just any 150
members of each race (as a class, and not the best I have known, nor the worst
pound units, but to the first 150 pounds on the scale. From the 'score' of 150,
members), to one or more of the classifications which I have circled.
we can reproduce exactly which units on the scale were covered and which
were not. This is not an essential requirement when we are dealing with
constant and interchangeable units, such as kilograms or inches; but if we
were dealing, say, with the symptoms of the different stages of an illness, it
would be helpful if they could be ordered or scaled in terms of their degree of
seriousness in such a way that the presence of symptom D would mean that
the patient also must have symptoms A, B, and C. Similarly, a score on an
attitude scale might show us, by means of a single figure, with which
statements the respondent agreed and with which he or she disagreed, thus
telling us the respondent's place on the attitude continuum. This is a
requirement that in practice is difficult to achieve, for many attitude pools are
not amenable to this kind of cumulative or progressive scaling - partly Canadians 1 2 3 4 5 6 7
because they may not be uni-dimensional. Ch'inese 1 2 3 4 5 6 7
English 1 2 3 4 5 6 7
Apart from these main requirements, there may be others. For instance, it is French 1 2 3 4 5 6 7
helpful to have norms or standards derived from the scores of large numbers of Germans 1 2 3 4 5 6 7
respondents, enabling us to compare an individual's score with those of others Hindus 1 2 3 4 5 6 7
and interpret its meaning. etc.
In the following sections we will discuss the four best-known methods of
attitude scaling, namely the Bogardus, Thurstone, Llkert and Guttman scales. Apart from the rather complicated instructions, at first sight there seems little
One might ask why more than one method is needed. This has come about to criticize in this scale. The idea of expressing ethnic prejudice as 'social distance'
because, since the 1920s, different research workers have developed methods of seems reasonable, and by analogy we may think of a straight road with the seven
scale-building in which they have laid particular stress on one or another of the classifications as milestones along it. Early large-scale investigations did show a
above requirements, and have paid less attention to the others. One method has fairly regular progression in the proportions of Americans endorsing each
concentrated on uni-dimensionality, another on finding equivalent scale units, a position. Usually, there is a 'hump' somewhere in the middle - suggesting
third on obtaining reproducibility and so on. There does not seem to be a method neither extreme hostility nor complete acceptance, depending on the 'race' or
that combines the advantages of them all, and it is therefore very important that nationality in question.
we understand their respective aims and the differences between them. How well does this scale satisfy the five requirements outlined before?
It follows that, for the present, it is impossible to say which method is best. Concerning uni-dimensionality, we have little evidence. It is quite possible that,
Each has important desirable features, but each is also open to criticism. The best underlying these various categories, there is a single, pure attitude, but then
method for any enquiry is the one which is most appropriate to the particular again, such categories as 'marriage', 'neighbours', 'exclude', and others may bring
problem. If we wish to study attitude patterning or to explore theories of in many extraneous influences. Criticism has been directed against the scale's
190 Questionnaire Design, interviewing and Attitude Measurement Attitude scaling 191
questionable linearity and its unequal intervals. For some respondents, admis- extremely unfavourable items will go into pile No. 11. The intervals between the
sion to occupationalmembership signifies a greater degree of closeness than 'to categories should be regarded as subjectively equal. It is most important that the
my street as neighbours', and so on. T h s would require a different (or perhaps judges ignore their own agreement or disagreement with the item; their job is
non-linear) ordering of the steps - though in large samples this might only merely to place the item in a particular category according to its meaning.
apply to a minority. As for equality of intervals, there is no evidence that the Irrelevant, ambiguous or poorly worded items will have been eliminated
social distance between, say, 2 and 3 is in any way equivalent to the distance beforehand. At the end of the judging process, each judge will record on each
between 5 and 6 and so on. The scoring is thus completely arbitrary. Reliability statement the number of the pile in which it has been placed.
of the scale has reached beyond .90 and must be regarded as satisfactory. There Next, we assemble all the judgments for each statement. If we consider a
is, at least, some positive evidence of the validity of the scale in comparison with hypothetical statement, No. 95, we will no doubt find that most judges have
other types of scale. Until lately, the question of reproducibility had not been placed it in, say, category 8, while some have placed it in 7 or 9, and a few in 6,
raised, but more recent inquiries have shown that this can be surprisingly good. 10,or 11. We are now going to consider the distribution of judgments for each
If we disregard the scoring and consider the results merely as a way of item in order to select, from our total pool of items, those two dozen or so that
ordering individuals or groups with regard to their ethnic attitudes, then useful will form a scale.
comparisons can be made, but a good deal of caution is required in the First of all, we consider the spread of judgments. The wider the spread, the
interpretation of such results. more ambiguous the item is. Ambiguous items will cause unreliability and have
no place in our scale; we therefore wish to eliminate them. Almost any measure
of spread (range, variance, standard deviation, semi-interquartile range) will do,
Thurstone scales but the last is probably the simplest and quickest, and at the same time it allows
Thurstone's chief preoccupation was with the problem of equal or, rather, us to read off the median.
equal-appearing intervals. Part of his reasoning stemmed from experiments in We need the medians in order to examine item position or scale value. After
psychophysics, where various methods have been developed to scale the eliminating the items with excessive spread, we will probably still have a
judgments of individuals with regard to various physical properties such as considerable number of statements left, and their medians or scale values will be
weight. The smaller the true difference in weight between two objects, the unevenly distributed along the hypothetical continuum of our scale, so that we
fewer people will be able to distinguish correctly between them (see Guilford, may get medians reading 1.7,1.8,2.2,2.2,2.9,3.0,3.1,3.1,3.2,3.2,3.4,3.5, and
1956 for an exposition of various psychophysical methods). In much the same so on. (Medians are used because in psychophysics a 'just noticeable difference'
way, Thurstone attempted to devise attitude scales by getting people to compare between two stimuli has traditionally been defined as one that could be
attitude statements two at a time and deciding which of each pair was the more distinguished by 50 per cent of the respondents. Thus, for example, if the
positive or the more negative. median calculated for one statement is 4.0 and for another 3.0, then the two
This method, known as the paired-comparisons technique, becomes very statements may be said to lie one just noticeable difference apart on a
cumbersome when more than fifteen or twenty items are involved, as is psychological continuum.) We have to decide how long our scale is going to be
frequently the case. A less precise, but also less laborious, method was therefore and how many items we need to select. Twenty or twenty-two items would be
proposed by Thurstone, known as the method of equal-appearing intervals. common practice, but longer and also much shorter scales exist; or we may
The first step in this and every other scaling procedure is to design and collect decide to build two parallel forms of the scale, if the available items permit this.
a pool of items (see Chapter 10)from the literature and from pilot interviews. The detailed steps of the Thurstone scaling procedure are as follows:
After that, we need a group of 'judges'. While in the past university students
were often used as judges, it is preferable to have the judgments made by people SAMPLE INSTRUCTIONS TO JUDGES
who are similar to those to whom the finished scale will be applied. To give some
rough idea of numbers: one might have 100-150 statements and 40-60 judges, 1. On the enclosed paper slips you will find a set of statements concerning
though as many as 300 judges have been used. . The statements are in no particular order, but they range from
Next, the items are reproduced individually on cards or slips of paper, and each extremely favourable to extremely unfavourable. Before starting the
judge is given, in random order, a complete set of items. The judges are told what judgment process, read through all the items carefully.
the scale is 'about', and a hypothetical scale is set up, running from 'most 2. Note that there is only one item on each slip, that the item number is on the
favourable' through a neutral point to'least favourable'. The scale is divided into left, and that the space for recording your judgment is on the right.
a fixed number of sections, usually eleven, and it will be the task of the judges 3. Sort out the items into eleven piles. In pile No. I, you should put those items
to consider each item and to place it in one of these eleven sections or piles. In pile that, in your opinion, would reflect the most favourable attitude to -. In
No. 1should go only the extremely favourable items, in pile No. 2 the slightly pile No. 2 will be those items that you would regard as slightly less favourable
less favourable ones and so on through pile No. 6, which should contain only to - . Thus each successive group of statements should be slightly less
neutral items. Slightly unfavourable items go into pile No. 7, and so forth; the favourable than the preceding one, until you reach pile No. 6, which should
192 Questionnaire Design, Interviewing and Attitude Measurement Atfitude scaling 193
contain those statements that show a neutral attitude. In pile No. 7 would go Table 11.1.
those items which you regard as slightly unfavourable and so on until you
reach pile No. 11,which should reflect only the most rejecting and hostile
Cumulative
attitudes in the item pool. Pile No. Frequency Percentage percentages
4. Your own agreement or disagreement with these items must be disregarded.
5. Write on each item slip, in the space provided, the number of the pile in which
you put it.
6. Feel free to use any of the eleven points on the scale, but do not attempt to
put the same number of slips in each pile.
7. After recording your judgments, put all the slips together into an envelope.
INSTRUCTIONS
FOR CALCULATION OF MEDIANS AND Q's
I. The median and the semi-interquartile range may either be calculated, or
can be 'read off' from a cumulative-frequency graph. In the latter case,
proceed as follows.
2. Sort all the items returned by the judges according to their item numbers.
Put all the item slips bearing the same number together.
3. For each group, sort the item slips into piles, according to the written
judgments recorded on each slip. You will find most often that you will need
from four to six piles - occasionally more. 10. For the example in Table 11.1, the following values have been found:
4. Draw up a frequency distribution for each item by writing down the pile median = 9.0
numbers from 1to 11and recording the number of slips in each pile. If there Q = ~ ( 9 . -8 7.8) = 1.0.
are no slips for some pile numbers (categories),record a zero.
5. Add up the total number of frequencies, i.e. the total number of slips you The actual selection of the items will be in terms of the intervals between
have for this item. Convert the frequencies for each category into them. This might be one whole scale point, half a point or any other distance. If
percentages of this total. we decide, for instance, that we want about twenty items, then we may want to
6. The distribution may now be made cumulative, by adding the percentages choose those items whose medians are nearest to scale points 1.5,2.0, 2.5, 3.0,
successively to one another. When the last percentage has been added, the 3.5,4.0, and so on. In this way we build up a scale with equal-appearing intervals.
cumulative percentage frequency should read 100. Table 11.1 may clarify In exceptional cases, we may wish for finer discrimination on a particular part of
this process. the scale than on the remainder, and we may choose items which lie closer
7. Draw up a cumulative graph for the cumulative-percentage frequencies. together on that section.
The eleven categories are placed along the bottom line, while the Our scale is now ready for administration. The statements will be printed in
percentages are indicated along the side. Find the approximate point directly random order, without their scale values. Respondents will only be asked either
above each category value that corresponds to the correct cumulative to agree or to disagree with each statement. Subsequently, we look up the scale
percentage for that category. Draw lines to connect these percentage value of each agreed statement, and the median of these constitutes the
points. This will give a steeply rising curve of cumulative-percentage respondent's score. If all goes according to plan, our respondent should only
frequencies. agree with a very few statements. If, let us say, he is a male with a mildly positive
8. Lines should be drawn across the page for the median (50 per cent of the attitude, then he should disagree with items that are strongly positive, neutral
total frequencies) and the two quartiles (25 per cent and 75 per cent or negative. In fact, ideally, he should agree only with the one or two items that
respectively). Their corresponding scale values may be read off by noting best reflect his particular attitude.
the points at which these lines cross the cumulative-percentage curve and Although, as we have seen, Thurstone's procedure is primarily concerned
dropping perpendiculars to the base-line at those three points (see Figure with locating items at points on a hypothetical scale, we must be careful when
-I?-I\
11.1). treating these scale values as actual numbers, as intervals that are additive and
9. Record the median from the value of the central perpendicular. Calculate interchangeable on a linear continuum. We are dealing not with equal, but with
Q = the semi-interquartile range as follows: record the scale value of the equal-appearing intervals, with psychological rather than numerical units, which
third perpendicular; subtract from it the value of the first perpendicular; may or may not be equal. Moreover, since the division of the continuum into
halve this difference to obtain Q. eleven units is arbitrary, we should not think of a score of eight as numerically
194 Questionnaire Design, Interviewing and Attitude Measuremenf Attitude scaling 195
known (because of their membership in certain organizations, their comments
in an interview, or their responses to essay-type questions), but since these
-80 other measures may be less reliable than the scale that is being validated, or may
8 0 - Third quar tile
refer to different attitude facets, doubts must remain.
7Or
- 70
Judges should be similar to those on whom the scale will be used. However,
- 60-
Medion
-60 '=
0
do different groups of judges produce the same scale values? For instance, would
'
e black judges produce the same scale values as white judges for an item pool
' 50'
40-
3 0 - First quartik
50
-40
- 30
';:
dealing with ethnic prejudice? By and large, experimental investigations have
shown that the scale values obtained from such widely differing groups of
judges correlate highly with one another so long as judges with extreme views
20 -
- 20 are excluded. Judges with strong 'pro' or 'anti' feelings concerning the particular
10 - -10 attitude in question tend to bunch the items toward the opposite end of the scale;
0. 1 I I I 1 0
thus, judges with strong pro-black attitudes tend to displace items toward the
1 2 3 4 5 6 7 8 9 1 0 1 1 anti-black end of the scale and to place few items in the middle (neutral)
categories. In practice, judges who place more than one-third of the items in the
Figure 11.1. Cumulative-frequency graph for item no. - same category should be excluded; inevitably, all judges will be influenced by
median = their own attitudes, but in most cases the effect will be small and will barely
Q = 1/, (third quartile minus first quartile) = affect the order of the items.
Caution is also required when scales are applied cross-culturally and when
some of the older scales are reapplied today - issues, words, and stereotypes
twice as high as a score of four, and so on. In any case, in scale construction it is sometimes undergo subtle changes (see p. 183).
often felt that more units would be preferable on one side of the neutral point
than on the other; some attitudes seem more differentiated at one end of the
scale, or go further or deeper, and judges often make finer discriminations at the
extremes of a scale than in the middle. Doubts have also been cast on the Likert scales
comparability of scores from one scale to another in a battery of several attitude
scales. The construction of a Thurstone scale always means a lot of work, and it is often
What can we say about the uni-dimensionality of Thurstone scales? Our first difficult to obtain an adequate group of judges. The Likert procedure may have
guide here, apart from the care taken in the composition of the item pool, is the its disadvantages, but it is certainly less laborious, and this - together with the
elimination of the more ambiguous items, the ones with high semi-interquartile experimental finding that Likert scales correlate well with Thurstone scales -
ranges. Items on which general agreement exists among judges have an a priori has helped to make it the most popular scaling procedure in use today.
claim for inclusion, but this is still quite removed from proven uni- Likert's primary concern was with uni-dimensionality - making sure that all
dimensionality. We can, however, go further and administer our newly the items would measure the same thing. He also wanted to eliminate the need
developed scale to a sample of respondents, with instructions to tick the for judges, by getting subjects in a trial sample to place themselves on an attitude
statements with which they agree. Next, we can analyse the results to check on continuum for each statement - running from 'strongly agree' to 'agree',
the internal consistency of the items. If respondents who agree with one 'uncertain', 'disagreeJ, and 'strongly disagree'. These five positions were given
particular statement also tick other statements with widely differing scale simple weights of 5,4,3,2, and 1for scoring purposes (or sometimes 4-0), after
values, the statement has to be rejected since it obviously contains large more complex scoring methods had been shown to possess no advantage.
elements that are irrelevant to the attitude that the scale should measure; if only To produce a Likert scale we proceed as follows: first, as usual, we compose an
statements with roughly similar scale values have been ticked, the statement can item pool. However, for the Likert procedure it is best not to have many neutral
be retained. (See also discussion of factorial scales later in this chapter.) items or many extreme items at either end of the continuum. Next, we need a
The reliability of Thurstone scales tends to be adequate, and they have the sample of respondents on whom to try out the entire pool of items together.
additional advantage that often a parallel form emerges from the item analysis; Probably 100 respondents would suffice for most purposes, but numbers of the
this is particularly useful when studying attitude change. Reproducibility (in the order of 250 or 300 are not unusual. Each respondent will be asked, not merely
technical sense) would, presumably, be good in the ideal case where a respondent whether helshe agrees or disagrees with each statement, but to tick one of the
endorses only a single item, but since this happens very rarely the scales may be five positions given above. Respondents should be similar to those on whom the
criticized on this account. The validity of these scales has occasionally been scale will be used.
demonstrated with the aid of criterion groups whose attitudes were allegedly Next, we score the record of each respondent. To do this, we must decide
196 Questionnaire Design, Intemiewing and Attitude Measurement
I Attitude scaling 197
'strongly agree' up to 5 for 'strongly disagree'. If we decide that a high score will
whether we want a high scale score to mean a favourable or an unfavourable
mean an unfavourable attitude, then the opposite system of scoring will apply.
attitude. It does not matter what we decide, but from then on we must be
It helps, therefore, if we have few neutral items, making it easy to tell from the
consistent. If we decide that a high score on the scale will mean a favourable
wording of the statement whether it is positive or negative. But if we feel
attitude, then favourable statements must be scored 5 for 'strongly agree', down
uncertain about some statements, we can score them arbitrarily from 1-5 or
to 1for 'strongly disagree' - and unfavourable statements must be scored 1for
from 5-1; the correlations will subsequently show us whether we are right.
Research workers often get into difficulties over this problem of scoring
Strongly Strongly reversals, so it is important to be meticulous about it from the start.
agree Agree Uncertain Disagree disagree Having scored each item from 1-5 or from 5-1, we next add up the item scores
2 1 to obtain a total score. For instance, if we have 132 items in our pool, then the
5 4 3
possible range of total scores will be from 132 to 660 (5 x 132) for each subject.
Figure 11.2 illustrates some items from a scale for mothers, dealing with
(I) Children bring a J acceptance or rejection of children. It becomes obvious, on reading through the
husband and wife closer
to each other.
items, that some of them express greater or lesser acceptance, others express
degrees of hostility or rejection, and one or two may not fit on this particular
(2) It is fun showing J dimension. Thus, agreement with statement (2)'It is fun showing chldren how
children how to do to do things' would seem to imply positive feelings for children; agreement with
things. statement (3)'Children need some of their natural meanness taken out of them'
(3) Children need some of J would seem to indicate hostility to children on the part of the respondent; while
their natural meanness the implications of statement (8)'If we could afford to do so, we would prefer to
taken out of them. send our children to a boarding school' are unclear: agreement might signify
(4) A mother with young rejection of the children, or it might mean the desire to lavish money on them
children badly misses J in order to give them a better education (which could, however, be a form of
adult company and overcompensation for unconscious feelings of rejection) (see page 163).
conversation. We now come to the problem of scoring. If we decide that a high scale score
(5) On balance, children J is going to mean a positive attitude to children, then agreement with the
are more of a blessing statements that imply love of children should be scored 5 or 4, and agreement
than a burden. with statements that imply rejection of children should be scored 1or 2 - in
4
other words, the scoring of these latter statements is reversed. If, on the other
(6) It is often difficult to
keep one's temper with
hand, we decide that a high scale score will mean a negative attitude to children,
a child. then the scoring on the items that imply positive feelings toward children (items
1,2,5,9 and 10)should be reversed. The important thing is to be consistent. We
(7) Looking after children J must also make sure that we write our scoring instructions correctly, and that
really demands too in the case of computer analysis of each statement, we know whether the
much of me.
(8) If we could afford to do
I( respondents' ticks were entered as given or were reversed, where necessary,
before data entry.
J
so, we would prefer to In Figure 11.2 we have given but ten items, and we have shown the responses
send our children to a of one particular mother. Just glancing at her responses we get the impression
boarding school. of a mother with a mildly positive attitude toward children (items 2,5,7 and 10)
(9) When things are who is able to express some moderately negative feelings (items 4 and 9) but
difficult, children are shies away from extreme hostility (item 3) or inspired love (item 9). She also
often a great source of expresses some doubts (items 1and 8); this may be because she is uncertain of
courage and inspiration. her feelings, or uncertain of the implications of the items. Perhaps these items
(10) If I had my life to live do not belong in this scale because they contain other powerful attitude
over again, I should J components (to marriage, to social class, t o boarding schools, to separation)
again want to have besides acceptance or rejection of children. Item analysis would show us
children. whether these items should be retained.
Figure 11.2. How should this mother's response be scored? Let us assume that we have
198 Questionnaire Design, Interviewing and Attifude Measurement Aftitude scaling 199
decided that a high scale score shall mean a positive attitude to children. In that membership. We must therefore say to ourselves that, for the moment at least,
case, all the positive items can be scored as they stand: the best available measure of the attitude concerned is the total item pool that we
have so carefully constructed. By purifying this, the items will at least be
5 = strongly agree consistent and homogeneous -they will all be measuring the same thing -and
4 = agree the scale may possibly also be valid.
3 = uncertain If we are prepared to make this assumption, then the rest is plain sailing. We
2 = disagree simply work out correlation coefficients for each item with the total score and
1= strongly disagree. retain those with the highest correlations. This is an internal-consistency
method of item analysis, since no external criterion is available. We should not
The scoring for items 3, 4, 6, 7 and 8 will, however, have to be reversed as really use the total score, but the total score minus the score for the item in
follows: question. This means that, for each item in turn, we will have a slightly different
set of total scores. However, this subtraction procedure will not often make
1= strongly agree much difference, especially if the item pool is at all large; many research workers
2 = agree do not bother with it.
3 = uncertain Here is an example. Suppose we are concerned with the analysis of Item 5
4 = disagree from the item pool in Figure 11.2. We have the scores of ten respondents on the
5 = strongly disagree. pool as a whole, on Item 5, and on the pool as a whole minus their score on Item
5 (see Table 11.2).
Note, by the way, that the scoring of 'uncertain' is the same (namely 3) in both We can now calculate the relationship between Item 5 and the total score (or
cases. We can now give a numerical score to each tick (or program the computer rather, between Item 5 and the total score minus the scores for Item 5). This
to do SO) and calculate a total score, as follows: relationship is very strong ( r = .96), and we should probably wish to retain this
item if such results had been obtained on the basis of a much larger sample and
item 1= 3 a large item pool.
2 = 4 Our final scale will consist of, let us say, eighteen items. They have no scale
3 = 5 values such as we find in the Thurstone procedure; all the items carry equal
4 = 1 weight. The respondents, as before, are asked to indicate their degree of
5 = 4 agreement, and these responses are subsequently scored 1-5, remembering that
6 = 4 scoring for some of the items may have to be reversed. After that, the eighteen
7 = 4 item scores are added up for a total score, which is the respondent's score on our
8 = 3 scale.
9 = 2
10 =>
total score 35
Table 11.2.
Since there are ten items, we have a maximum possible score of 5 x 10= 50, and
a necessary minimum score of 1 x 10= 10. A score of 35 is thus a little above the Total Score on Total score
midway point toward the positive end of the scale -which confirms our earlier Respondent score item 5 minus item 5
impression of someone with mildly positive attitudes towards children.
Now we shall want to carry out an item analysis to decide which are the best
statements for our scale. Ideally, the item analysis should take place by
correlating each item with some reliable outside criterion of the attitude that it
is supposed to measure, and retaining only the items with the highest
correlations. Such external criteria are, however, almost never available. It
would not be safe to infer from the fact that a woman has children that she
necessarily loves them; nor can we be sure that people who vote for a certain
political party necessarily occupy a given position on a political spectrum; or that
professional military men are necessarily more war-minded. In other words, it
is dangerous to infer people's attitudes from their behaviour or from their group
200 Questionnaire Design, Interoiewing and Attitude Measurement Attitude scaling 201
Some investigators have used a seven-point degree-of-agreement score Factorial scales (see Chapter 9)
rather than the more usual five points. Others prefer to use some other index,
rather than correlation coefficients, for item selection. In the psychometric Internal-consistency methods of item analysis provide some safeguard against
literature there are methods of item analysis, many of which could be applied to the inclusion of unrelated items in a scale, but clearly a better way of ensuring
attitude scaling. uni-dimensionality would be through the use of factor analysis. In the case of a
Reliability of Likert scales tends to be good and, partly because of the greater set of attitude items or statements, when we are looking for a single score to
range of answers permitted to respondents, is often higher than that of express an individual's position on an attitude continuum, we will want to use
corresponding Thurstone scales; a reliability coefficient of .85 is often achieved. factor analysis for scale-building purposes, in order to eliminate items that do
The scale makes no pretence at equal-appearing intervals but by using the not belong and to keep items that have high 'loadings' on the factor (attitude)
internal-consistency method of item selection it approaches uni-dimensionality that we want to measure. Sometimes, however, we can use factor analysis to
in many cases. The number of items in a Likert scale is arbitrary but is sometimes show how a seemingly homogeneous attitude complex in fact 'breaks down'into
very small. several independent factors. A person's political orientation, for instance, has
The most serious criticism levelled against this type of scale is its lack of been shown by Himmelweit et al. (1981) to consist not merely of a left-wing1
reproducibility (in the technical sense): the same total score may be obtained in right-wing dimension, but of several other independent dimensions. Parental
many different ways. This being so, it has been argued that such a score has little childrearing attitudes have similarly been shown to have at least two independ-
meaning or that two or more identical scores may have totally different ent components (see also the androgeny scale items, p. 181).
meanings. Often, for this reason, the pattern of responses becomes more Factor analysis is a laborious technique unless one has a computer available.
interesting than the total score. Often, therefore, we find factor analysis applied not to the original item pool but
Another criticism has been that since the scale offers no metric or interval to finished scales. When applied to a single scale, it can throw light on its uni-
measures and lacks a neutral point, we do not know where scores in the middle dimensionality after all the individual items have been intercorrelated. When
ranges change from mildly positive to mildly negative. As against this, it should applied to a battery of scales all given to the same respondents, it can reveal
be pointed out that percentile norms or standard-deviation norms can be similarities and differences between the scales as well as some of the underlying
calculated if a sample of sufficient size is available; certainly, Likert scales will attitudes or value systems. Factor analysis then becomes a tool for theoretical
effectively separate people within the same group. With regard to the neutral investigation and new discoveries. In this connection, the problem of rotation
point on the scale, we must agree that this is not necessarily the midpoint and naming of factors remains a source of difficulties (see p. 170).
between the two extreme scale scores. Moreover, scores in the middle region Factor analysis opens the way to other possibilities. In the rare instances
could be due to lukewarm response, lack of knowledge or lack of attitude in the where an external criterion is available, validation and purification will obviously
respondent (leading to many 'uncertain' responses) - or to the presence of both lead to a very high-quality scale. More refined scoring might sometimes be
strongly positive and strongly negative responses which would more or less useful, by weighting the items according to their factor loadings. Factor analysis,
balance each other. With such different possibilities, the neutral point would be even more than the original Likert scaling procedure, enables us to show the
difficult to locate and even more difficult to interpret. common attitudinal basis for some widely diverse issues. For example, it has
In practice, if we remember that equal score intervals do not permit us to make been shown that opinions on sovereignty, abortion law, corporal punishment
assertions about the equality of underlying attitude differences, and that and religious education have a common underlying attitude structure (Eysenck,
identical scores may have very different meanings, the Likert scales tend to 1954).
perform very well when it comes to a reliable, rough ordering of people with The factor analysis of attitude scales also opens the way to cross-national
regard to a particular attitude. Apart from their relative ease of construction, comparisons. As a rule, these present problems because of the difficulties of
these scales have two other advantages. First, they provide more precise translation and 'equivalent meaning', and because one never knows whether the
information about the respondent's degree of agreement or disagreement, and attitude concerned is structured in the same way in another country. However,
respondents usually prefer this to a simple agreeldisagree response. Second, it if scales are translated and factor-analysed in several countries and continue to
becomes possible to include items whose manifest content is not obviously produce similar factor pictures, then this strongly suggests similarity of attitude
related to the attitude in question, enabling subtler and deeper ramifications of structure and opens the way to cross-national comparisons (see page 184).
an attitude to be explored. These 'long shots', such as the item about sending
children to boarding schools in our earlier example, which are unlikely to be
included in the Thurstone procedure, enable us to make use of the links that an Scalogram analysis
attitude may have with neighbouring areas and to uncover the strands and
interconnections of its various components. The chief concerns of the method of scalogram analysis, developed by Guttman
and his associates, are the twin problems of uni-dimensionality and reproducibil-
ity. The method enables us, from a respondent's score, to know exactly which
202 Questionnaire Design, Interviewing and Attitude Measurement Attitude scaling 203
items he or she has endorsed, with not more than 10 per cent error for the procedures of scalogram analysis are designed to test a given universe of content
sample as a whole. This is, of course, in marked contrast to the Likert scale or, at any rate, a group of items for 'scalability' by seeing whether they will yield
scores, which can be arrived at in a number of different ways. The method can a scale with a satisfactory coefficient of reproducibility (in practice, a coefficient
be used with items other than attitude statements, such as neurotic symptoms of .90 or over is accepted).Of course, not all areas of content will scale, especially
or the possession of consumer durables. not if they are rather wide and heterogeneous, and one cannot know beforehand
The items in a Guttman scale have the properties of being ordinal and whether the attempt at scale construction will be successful. The procedure has
cumulative. For instance: lead, glass, and diamond are ordered according to their been criticized for this and for a tendency to produce scales covering a very
cumulative degrees of hardness; addition, multiplication and the extraction of narrow universe of content (because wider areas often do not scale).
square roots are arithmetical operations ordered according to their cumulative The procedure of scalogram analysis is designed to enable us to see how far
degree of difficulty (it is highly likely that anyone who can multiply can also add our items, and people's responses to them, deviate from the ideal scale pattern.
and that anyone who can extract square roots can both add and multiply).If we Suppose that we start out with a set of social-distance items and that we have
think of a dozen or more degrees of hardness or difficulty, ranked in order, then given these (as yet unscaled) items to several hundred respondents. A tabulation
many respondents will endorse the early ones -indicating that they know how of the results for the first fifteen cases might look llke Table 11.3.
to add, subtract, multiply and so on - but, sooner or later, they will 'cross over' Note that we have given each respondent a score equal to the number of yes-
and fail to endorse such remaining items as solving differential equations or answers given. Let us now rearrange the table, by placing the rows (respondents)
carrying out integrations. This cross-over point is their individual score. From it, in order of their scale score. The result is shown in Table 11.4.
we know precisely which items they must have endorsed. Next, let us rearrange the columns (items)in the table, by putting them in order
Another illustration might be a social-distance scale. If it has been constructed of the number of yes-answers given to each item, as shown in Table 11.5.
with the aid of scalogram analysis, then the individuals' scores will tell us which We have now produced a scalogram pattern that is very nearly perfect and has
items they have endorsed, since the score tells us at what point they have crossed very high reproducibility. A score of 3 clearly means a yes-response to items 7,
over to a negative attitude. 5, and 1(and not to any three items, as would be the case in a Likert scale); a score
Over the years, many Guttman scales have been developed successfully, some of 6 means a yes-response to items 7,5,1,8,2 and 4; and so on. The lowest item
in the socio-political field, others in psychiatry, in consumer research and so on on the scale is item 7, to which nearly everyone agrees; the highest or, in this
(see, for example, Marsh, 1977). case, most intimate one is item 3, to which almost no one agrees. Note the
Guttman set great score by the property of reproducibility, which, as we have
seen, is a useful but by no means necessary attribute for most scales. The
Table 11.4.
Table 11.3. Item Item Item Item Item Item Item Item Score
Respondent 1 2 3 4 5 6 7 8
Item Item Item Item Item Item Item Item Score
Respondent 1 2 3 4 5 6 7 8
Table 11.5. recalculate the coefficient of reproducibility. In the above example, for instance,
we might wish to drop item 8 (which contains one error, and has two gaps
Item Item Item Item Item Item Item Item Score
caused by other errors), and item 3 (which is so extreme that it fails to
Respondent 7 5 I 8 2 4 6 3 differentiate, and also because it contains one error).
Scalogram analysis is usually carried out on one or more samples of a hundred
- cases each. The number of items might initially be several dozen that, after
7 yes yes yes yes yes 7
- 7
repeated attempts at scaling, might be whittled down to perhaps eight or ten
9 yes yes yes yes yes yes
items which form a scale. Such a scale might sometimes be much narrower in
,
10 yes yes yes yes yes - 6
I content than the original item pool; at other times, seemingly unrelated items
1 yes
yes
yes
yes
yes
yes
-
yes ;:;
yes yes
yy 1
-
-
yes
-
6
6 are unexpectedly easy to scale - there is no way of forecasting the outcome. If
::: 1
yes yes yes yes - - 5 we have many respondents, it would be advisable to repeat the scaling operation
2 yes yes yes - - - - 4 on several samples of one hundred, to reduce the possibility that a scale pattern
6 yes yes yes - - - - 4 is improved by a combination of chance variables. We must also guard against
8 yes yes yes - - yes - - 4 spuriously high estimates of reproducibility by first omitting all items to which
almost everyone or almost no one agrees; usually, therefore, items with less
than 20 per cent or more than 80 per cent endorsement are omitted from a
scalogram analysis at the start.
So far, we have assumed all the items to be dichotomous. However, we might
have items with multiple responses such as degrees of agreement, or frequency
of a symptom. Many investigators would probably fix a cut-off point for each
item and turn them into dichotomies. However, the place of the cut-off point
triangular pattern of the responses -this is the pattern that scalogram analysis will affect the scale position of the item, and a certain amount of trial and error
aims to produce. may be necessary before we find the cut-off point that produces the least
We should observe several further points. First of all, Table 11.5 is a 'rigged' number of scale errors. It is also possible to treat each of the multiple responses
example for demonstration purposes; in practice we may have to rearrange the as a single dichotomous item; thus, if we had two items, one with three response
order of items and of respondents not once or twice, but many times, until we positions and another with five response positions, we might treat these as eight
have obtained the nearest approximation to a scalogram pattern. This procedure separate items and try to scale them as before. Or we might wish to group the
becomes very tedious and fraught with errors, especially if we deal, not with response positions by turning, say, five responses into three. Obviously, this
fifteen respondents, but with a hundred or more, and with dozens of items kind of analysis will increase the amount of work involved in the scaling
instead of a mere eight. However, modern statistical software packages operation.
frequently contain a scalogram analysis program. These should make Guttman Guttman has been criticized for his insistence on reproducibility, and for
scaling more accessible. The researcher can also quickly see whether or not a regarding cumulative reproducibilityas the chief criterion for a'truel scale; other
particular domain will 'scale1, in Guttman terms. writers have felt that these are valuable but not essential properties. He has also
Looking again at Table 11.5, we note that it is not absolutely regular. For been criticized for his somewhat arbitrary standards, such as the lower limit of
instance, respondent 8 has given a yes-response to item 4 instead of to item 8. 0.9 for coefficients of reproducibility. His procedures are laborious, and there is
Similarly, respondents 1and 11 show slight deviations from the perfect scale no certainty that, in the end, a usable scale will result. On the other hand,
pattern, and to this extent they make reproducibility less certain. Such scalogram analysis will prevent us from building a single scale for a universe of
deviations are called 'errors' in this context. The formula for calculating the content that really demands two or more separate scales; in other words, it
reproducibility coefficient is as follows: offers the important safeguard of uni-dimensionality.
Scalogram analysis can produce some short yet highly effective scales. The
no. of errors problem of validation remains and depends largely on the manifest content of
R=1- the items; also, the scales do not have equal or equal-appearing intervals, but as
no. of responses
a rule they are highly reliable.
(The number of responses is the number of items multiplied by the number of
respondents.) If reproducibility is below 0.9 then the scale is considered The problems of validation
unsatisfactory. In that case, we might have to give up the scaling attempt
altogether, or we might - as with the other scaling procedures - get rid of a In Chapter 8 we noted the difference between factual and attitudinal measures
number of items and then rescore and rescale the remaining responses, and and the greater difficulty of validating the latter because of their abstract and
206 Questionnaire Design, Interviewing and Attihde Measurement Attitude scaling 207
indirect nature, and because of the absence of suitable criteria. Attitude scales Yet attitude scales still have many uses, at least until some better methods
share this problem with other forms of mental measurement. The literature come along. The advent of modern computer software packages has given
contains but a small number of attempts at direct validation against a criterion, attitude scales an extended life-span. Where, in the past, researchers sometimes
and we may well ask whether the measures employed as criteria were shied away from scaling procedures because of the monumental amount of
themselves valid. Such attempts have included the use of essay-type questions, tedious arithmetic involved, they can now rely on tireless computers to carry out
experts' judgments, membership in groups with known policies or interests, internal consistency checks, to produce correlation matrices and factor analyses,
pictorial material, interviews and case studies, judgments by friends or co- and to attempt scalogram analyses in even the most recalcitrant domains. This
workers, self-ratings, political votes, and such overt behaviour as church has generally improved the quality of survey research, and the Likert five-point
attendance. New scales are often correlated with older, well-known scales format has become widely accepted. Many researchers will, however, not follow
which, however, may themselves be of questionable validity. Scales are often the Lkert procedures step by step, but will use whatever techniques of item
given names or labels that help to create a spuriously high impression of validity. analysis they find convenient. They will probably assemble their item pool under
The very fact that they look like tests and can be scored may create expectations several area headings or sub-domains, each dealing with a different aspect of the
of validity and exactitude that may not be well-founded. topic under investigation; they will probably not bother with item-total score
It may be helpful to remind ourselves of the different approaches to the correlations but will go straight on to do a large factor analysis. This will enable
problem of validity. We have repeatedly pointed out the weaknesses in the them to discard unwanted items and will produce a scale with high reliability. It
criterion-group approach, sometimes known as pragmatic validation. As we saw will also reveal some of the interconnections between the various aspects or sub-
in Chapter 9, for more theoretically oriented research the concept of construct domains, some of which may be useful for future research. Should time
validity has been developed, which hinges on the relationship of our scale to pressures make it impossible to carry out the necessary scaling work before the
other measures. We also saw that much depends on the quality of the attitude main field work, then some researchers will administer the entire item pool to
statements and the feelings that they arouse in the respondents; in this sense, the full survey sample (at some cost, no doubt), in the knowledge that the scaling
validity depends on the respondents' candour and willingness to cooperate and can be done by the computer subsequently, and will be all the more reliable for
the absence of stereotyped answers or 'facade' responses. Some investigators being based on larger numbers.
simply state that what the scale measures is indicated by the manifest content of Where computers offer no short-cuts is in the preliminary work of
the items; others rely on groups of judges for ascertaining what the items conceptualization,depth interviewing, pilot work and pre-testing. Now that for
measure. Of particular importance is predictive validity, usually in the sense of many investigators the scaling calculations present few problems, it is this
predicting some future aspect of behaviour such as voting (see Himmelweit et al, preliminary research investment which rightly remains the crucial requirement
1981, for a sophisticated approach to this problem). We can see from this that a of a good attitude scale.
great deal depends on our purpose in building a scale. It is one thing to require
a purely descriptive device, which can roughly divide our sample into several
groups with regard to a given attitude, but quite another to ask for a technique Selected readings
that will predict people's actions at some time in the future. Speaking very
generally, many of our scales will probably do an adequate descriptive job, as
ONATTITUDE SCALING THEORIES
long as not too much precision is required, but the problems of predictive validity
are a long way from being solved. Dawes, Robyn M., 1972, Fundamentals of Attitude Measurement, Wiley.
An introductory text, see especially sections on the Bogardus scale and on the
semantic differential.
General comments and evaluation Fishbein, Martin (ed.), 1967, Readings in Attihde Theory and Measurement, Wiley.
This very useful collection contains many classical papers dealing both with
Most of the theoretical underpinnings of attitude scaling go back to before measurement and with theory.
World War 11, and they owe more to psychometrics and educational testing Edwards, A.L., 1957, Techniques of Atfifude Scale Consfrudion, Appleton-Century-Crofts,
traditions than to modern social psychology. In theoretical as well as practical New York.
terms, these early techniques, and the linear-scaling model itself, have been
outstripped by subsequent developments in attitude theories, qualitative Cronbach, Lee J., 1990, Fifth Edition, Essentials of Psychological Testing, Harper and Row,
New York.
measurement, social-representations theory, ethnomethodology, studies of
Standard text on measurement and test construction. See discussion of item analysis
intentionality and so on. These later developments try to do better justice to the and scale construction, percentiles, reliability measures and validation.
subtlety and complexity of many attitudes, to their oblique yet powerful links
with the many interactive forces that produce overt behaviour, and to their McKennell, Aubrey C., 1974, Surveying Attitude Structures, Elsevier, Amsterdam.
distinctive place among the components of human personality. For the specialist.
208 Questionnaire Design, Interviewing and Attitude Measurement Attitude scaling 209
SOCIAL-DISTANCE
SCALES
Bogardus, E.S., 1933, 'A social-distance scale1, Sociological and Social Research, XVII, 265-71. Shaw, M.E. and Wright, J.M., 1967, Scales for the Measurement of Attitudes, McGraw-Hill,
Source publication for the Bogardus scales. New York.
See also Fishbein (above),chapter 9. An older collection, containing many Thurstone and Likert scales from the 'thirties
and 'forties, as well as a description of scaling methods.
THURSTONE
SCALING ESRC Survey Archive, University of Essex, Colchester, Essex C 0 4 3SQ, Great Britain.
Has extensive computerized holdings of British and some European survey
Thurstone, L.L. and Chave, E.J., 1929, The Measurement of Attitudes, University of Chicago materials. These include the General Household Survey from 1973. and a major
Press, Chicago. series of the Eurobarometer, the comparative polls held in the European Commun-
Source book for Thurstone scales. ity member countries.
See also Fishbein (above),chapter 10.
Social and Community Planning Research, annual, British Social Attitudes, Gower,
Guilford, J.P., 1956, Psychometric Methods, McGraw-Hill, New York. Aldershot, Hants.
An older reference book, but see especially chapters 4-7 on psychophysical methods. Regular series of detailed reports on British political and social attitudes.
Young, Copeland H., Savola, Kristen L. and Phelps, Erin, 1991, Inventory of Longitudinal
LIKERTSCALING Studies in the Social Sciences, Sage, London.
Source book of longitudinal studies in the United States and Canada going back sixty
Likert, R., 1932, A Techniquefor the Measurement of Attitudes, Columbia University Press, years.
New York.
Source book for Likert scaling. Robinson, John P., Rusk, Jerrold G. and Head, Kendra B. (eds), 1969, Measures of Political
See also Fishbein (above),chapter 11. Attiludes, Survey Research Center, ISR, University of Michigan, Ann Arbor, Mich.
Robinson, John P. and Shaver, P.R., 1973, Measures of Social Psychological Attitudes, Survey
Research Center, University of Michigan, Ann Arbor, Mich.
FACTORIALSCALES
Stewart, B., Hetherington, G. and Smith, M., 1985, Sumey Item Bank, MCBlUniversity
Eysenck, Hans J., 1954, The Psychology of Politics, Routledge & Kegan Paul, London. Press, London.
An early factor-analytic approach to attitude scaling.
Sudman, Seymour, and Bradburn, Norman M., 1983, Asking Questions, Jossey-Bass, San
Torney, Judith V., Oppenheim, A.N. and Farnen, Russell F., 1975, Civic Education in Ten Francisco.
Countries, Wiley. Lists the major US archives of survey data and questionnaires in chapter I .
Shows the uses of factorial attitude scales in cross-national research with children.
GUTTMANSCALING
Guttman, Louis, 1950, 'The basis for scalogram analysis1in Stouffer, Samuel A. (ed.),
1950, Measurement and Prediction, Princeton University Press, Princeton.
The main source book for the Guttman scaling technique and its theoretical
background.
See also Fishbein (above),chapter 12.
Marsh, Alan, 1977, Protest and Political Consciousness, Sage Library of Social Research, Vol.
49 Sage, London.
Developed a Guttman-type scale of Protest Potential.
VALIDATION
PROBLEMS
Ajzen, Icek, 1988, Attitudes, Personality and Behauiour, Open University Press, Milton
Keynes.
Deals with attitudinal consistency, prediction of behaviour and action intentions.
Himmelweit, Hilde T., Humphreys, Patrick, Jaeger, Marianne and Katz, Michael, 1981,
How Voters Decide, Academic Press.
Detailed attempt to predict voting behaviour.
Projective techniques in attitude study 211
devices. Such techniques should be used at first in a purely exploratory way;
later, they may be built and utilized to test specific hypotheses emerging from
the depth interviews.
PROJECTIVE
TECHNIQUES IN
ATTITUDE STUDY What projective techniques can do
When suitably designed, projective techniques can help to penetrate some of the
following barriers:
1. The barrier of awareness People are frequently unaware of their own motives
and attitudes and cannot give us the answers we need, even with the best will
in the world. For instance, very submissive people may keep dogs in order to
exercise their desire for dominance and control; lonely people may keep cats
Attitude scales rely for their effectiveness on the cooperation and frankness of as some kind of substitute for children; but these respondents might be quite
the respondent. This may have its disadvantages. For one reason or another, a unaware of such needs and motives.
respondent may try to fake or give a great many 'uncertain' responses. Fear, 2. The barrier of irrationality Our society places a high premium on sensible,
misunderstanding, the desire to place oneself in a more favourable light, social rational, and logical behaviour. Most of us tend to rationalize a lot of the time:
taboos, dislike for the research worker and other motives may all play a part in we stress or invent sound, logical reasons for actions whose origins are far
distorting the results and may lead to outright refusal. This has led to the from rational. For instance, if a sample of motorists is asked to rank the ten
development of methods of attitude measurement whose purpose is less most important characteristics of a new car, they will generally put 'reliability'
obvious to the respondent. and 'safety' high on the list, with 'styling' or 'appearance' toward the bottom.
Sometimes, by means of indirect methods, we can approach a deeper level in However, if they are asked to state which characteristics their friends or
the respondent than we can with attitude scales alone. For this reason, we may neighbours regard as important, 'styling' jumps to third or second place. The
wish to use both methods in the same inquiry to obtain data on the same average male driver is not aware of the extent to which he actually is
attitude-complex at several different levels. It is important to plan such an influenced by appearance - he likes to think that his choice is determined by
investigation with great care, or the results will tend to be contradictory instead such rational aspects as safety, performance, reliability and economy.
of complementary. In an experiment carried out at an exhibition, people were asked to choose
between two cameras in terms of quality. A considerable majority chose
camera B. Actually, the two cameras were identical, but camera B had a small
When to use projective techniques weight concealed inside it. Feeling the extra weight, people tended, subcons-
ciously, to think of that camera as a more 'solid job', with 'better materials' and
In planning our investigation, one of the first things we will have to decide is so forth. Yet if they had been asked for their reasons, they would no doubt
how deeply we need to probe. If we can stay at a relatively superficial level, then have spoken of shutter speeds, lens quality or flash synchronization, and any
direct techniques for attitude measurement such as the various types of attitude suggestion that weight played a part in their decision would have been
scales, ratings and rankings, grids and indices (see Chapter 13) can be used with regarded as preposterous.
advantage and will yield quantitative results. If, however, we have to penetrate 3. The barrier of inadmissibility Our society sets many norms and expectations for
deeper, perhaps below the level of conscious awareness or behind the all of us, and we find it difficult to admit to a stranger, or even to ourselves,
individual's social fagade, then indirect, projective techniques have to be used. that we sometimes fail to meet such standards. Ideally, we ought always to
Projective techniques can be particularly useful in evoking and outlining pay our fare on the bus, share our sweets, never buy pornography, tell only
stereotypes, self-images and norm-percepts; for instance, ideas connected with innocent jokes and drop our litter in litter-baskets - yet we sometimes fall
'the good husband' or 'the single parent'. short of these ideals, and this produces guilt feelings and a desire to 'cover up'.
The best way to find out whether or not we will need to probe in depth is to For instance, when passing around a box of assorted sweets, many people
carry out a series of careful depth interviews to explore the origins, complexities, have some lund of strategy that will prevent their favourite flavour from
motivational links and ramifications of the behaviour and attitudes in question, being picked by someone else - but one would need indirect techniques to
to enable us to conceptualize them. Extended interviews will most likely yield a find this out.
number of promising lines for investigation which may be studied by means of 4. The barrier of self-incrimination This is really a variant of the barrier of
attitude scales and other direct techniques, but which may also call for projective inadmissibility. It concerns those aspects of behaviour and feelings that might
212 Questionnaire Design, Interviewing and Attitude Measurement Projective techniques in attitude study 213
lower the respondent's self-esteem, such as ethnic prejudice, violation of officers: 'Why are uniforms worn?' Now, it is not at all clear whether the
sexual taboos, irrational fears, or alcoholic overindulgence. For instance, question refers to the uniforms of the wardens, or the uniforms of the
when men's toiletries (such as after-shave lotions and deodorants) were first prisoners or any other uniforms. Nor is it always clear what is meant by a
introduced, many men had the secret fear that they would be considered uniform; in some prisons, wardens wear ordinary clothes, made to a standard
homosexuals if they used these products. People about to undergo an pattern inside the prison, and these may or may not be considered 'uniforms'.
operation are sometimes extremely afraid that they will 'say terrible things' When respondents begin to answer such a question, they will almost
while under the anaesthetic. People do not admit such secret fears because unwittingly assign a particular meaning to it, and in so doing they have
they are afraid to make themselves look silly, childish, deficient or prejudiced, revealed their particular way of looking at the issues involved.
but projective techniques can often help us to uncover such feelings, attitudes 4. Conceptualizing We can find out something about respondents' attitudes by
or emotions. the way in which they name things, order things or group things. For
5. The barrier of politeness People often prefer not to say negative, unpleasant or instance, is insurance regarded as part of the concept 'investment'? Are Asian
critical things unless they have specific complaints. They tend to say that people regarded as 'black'? We can give respondents various labels or objects
most things are 'all right', and present a rather bland fagade. Thus, projective to 'sort in whatever ways you think they belong together' and question them
techniques could help to show us whether, say, a given newspaper was about the grouping afterward.
considered rather vulgar by certain groups of respondents.
All these approaches have certain elements in common. They all rely on
It follows that whenever the pilot work shows us that any of these barriers spontaneity of interpretation, on the fact that the respondent (and often the
exist with respect to the problem in hand, then the use of projective techniques interviewer) must not know the purpose behind the task or the questions. They
should be considered in the survey that is being designed. all rely on a greater or lesser degree of ambiguity in the task, or in the stimulus,
so that the respondents have to provide their own interpretations. They
should always be as nondirective as possible, in order not to bias the responses.
How indirect techniques work And they all require the making of inferences on the part of the research
worker.
There are four commonly used approaches, which overlap to some extent: In dealing with projective material, we are almost inevitably involved in
interpretation of the responses; that is, we do not necessarily take them at their
1. Association The 'say-the-first-thing-that-comes-into-yourmind' approach is face value but impose some significance or categorization of our own on them.
based on the assumption that a fast response to a stimulus word, picture or For instance, a small boy of ten was given a set of incomplete sentences to finish,
question will be less 'guarded' and therefore more revealing of underlying among them the sentence beginning: 'In the dark . . .'. This is what he wrote: 'In
attitudes and motives. the dark I am never, NEVER afraid!' Now it is clear that we should not simply
2. Fantasy The respondents are asked to guess, or tell a story, or discuss a accept this young m a n ~ u b l underlined
y assurances; indeed, we may well feel
picture in imaginary terms. In doing so, they have to use their own attitudes that he 'doth protest too much' and that he is probably very fearful in the dark,
and experience as 'building blocks' and this will give us an insight into some though he denies ths. We have here a problem in interpretation, and much
of the deeper levels of their personality. For instance, we might show a rather depends on how far we are prepared to go. At the completely 'objective'level, we
vague picture of a man with a worried face leaving a bank. The respondents' should classify the response as 'not afraid'- which would probably be nonsense;
task is to tell us what the man is feeling, and why, and what has taken place at a more interpretative level, we might classify it as 'denial of fear'; at a deeper
in the bank. In doing this, the respondents are likely to reveal some of their level still, we might simply classlfy it as a 'fear response'. The deeper we go, and
own attitudes and feelings toward banks and to saving, borrowing and the more we indulge in such interpretations, the less objective we become -and
financial problems generally. probably reliability (consistency between interpreters) also suffers. In the
3. Ambiguous stimuli All perception involves a certain amount of projection and construction and use of projective techniques we seek, therefore, some kind of
interpretation on the part of the respondent - the more so when the half-way house: a system of coding, scoring, classifying or interpreting that
stimulus is indefinite and ambiguous. The term 'projection' is not used here satisfies the demands of consistency and of objective scoring criteria, yet one
in the strict Freudian sense; rather, it refers to the fact that whenever that permits us to go some part of the way beneath the prima facie value or
subjects are asked to respond to a relatively ambiguous stimulus they will meaning of the responses.
reveal something about themselves when making their responses. Usually, There is often a painful conflict between the demands of objectivity, scientific
the subjects are not aware that the stimulus is ambiguous; to them, it has but method and rigour, on the one hand, and the desire to get the fullest flavour of
one clear meaning, and they proceed to respond accordingly. However, in meaning and significance out of such self-revealing responses, on the other. We
imposing their particular connotation on the stimulus they have already told have not yet acquired methods that will allow us to do both. In malung the choice
us something about themselves. For instance, we might ask a group - of -prison
of 'level', the social scientist faces a difficult task requiring judgment and complex
214 Questionnaire Design, Interuiewing and Atfifude Measurement Projective fechniques in aftifudestudy 215
analysis. Small wonder, then, that some of us build our half-way houses a Little However, in social research we tend more often to deal with general attitudes,
further down the slope than others. values, beliefs and feelings. For this reason, social investigators usually compose
their own incomplete sentences, tailored to the needs of their particular
problems.
Types of techniques It should be realized that sentence beginnings can vary greatly in their
ambiguity. Compare, for instance: 'if only . . .' and 'the nicest cigarette of the
Indirect or projective techniques have a long history in the field of clinical day . . .'It is clear that responses to the first item could range very widely indeed,
psychology, and some of the devices to be discussed in this chapter are simply whereas responses to the second item are limited to smoking and probably to the
applications or offshoots of early clinical tests. However, systematic attempts to mention of a particular time or occasion during the day.
make such tests valid, reliable and scorable have met with such great difficulties Sentence beginnings can also differ in subtlety. Some of them approach the
that most of them have fallen into disuse. Some remain valuable in the clinical problem in a highly predetermined way and obtain useful but rather narrow
field, where one is dealing with individual cases and where the test material can results. Others are more 'open' and their aim is less obvious; as a consequence
be used to gain further insight into the patient's problems, even if it does not the results may produce more spontaneous and more revealing information.
meet the demands of objectivity. In group situations, however, or where we Often it is best to have items of both types in a judicious mixture. Extensive pilot
have no supportive sources of information, such tests are found wanting. work is essential to make sure that the desired results are obtained.
Projective devices often present respondents with an ambiguous stimulus, In all social-research techniques, and in sentence completion in particular, the
which they have to interpret; in doing so, they reveal to us something of their effects of contexf should be kept in mind. For instance, if a respondent had been
perceptual world, their fantasies, their characteristic modes of responding, their asked a lot of questions about smoking habits and had been given a series of
frames of reference. Some stimuli, however, are more ambiguous than others. incomplete sentences dealing with cigarettes, the inclination would be to
One could imagine a gradient, all the way from such highly ambiguous stimuli complete the sentence 'If only . . .' with some comment about smoking. In this
as a cloud, a blank page or an inkblot, through rather more structured but still way, contextual influence can be a help or a hindrance, depending on our intent.
largely ambiguous stimuli such as incomplete sentences or uncaptioned If we had wanted to explore the respondents' more general worries and regrets,
cartoons, to highly structured stimuli that leave the respondent little latitude. It not only those linked to smoking, then the context would have produced a
has been suggested that such a gradient would roughly correspond to greater or rather misleading restriction, making it appear as if their chief concern in life was
lesser 'depth' within the person; the more ambiguous the stimulus, the deeper with cigarettes and smoking. On the other hand, we can sometimes utilize
the level of personality that it reveals - but the interpretation also becomes contextual influences of this kind to guide the responses into certain areas, but
more difficult and the problems of objectivity and validity are greater. In social this will have to be taken into account when we interpret the results.
research we have tended, therefore, to work with techniques of moderate In sentence-completion techniques we are loohng particularly for sponta-
degrees of ambiguity; many of our problems do not require the exploration of neity. When we seek to test a particular hypothesis or explore a certain problem,
the deeper levels of personality, and by using stimuli that retain some of their we tend to have greater confidence in a set of responses given to items that do
structure we can usually devise ways of making them suitable for large-scale not reveal our purpose. For instance, suppose we were conducting an
investigations. investigation into the fears and worries of hospital patients about to undergo
Before discussing some of these techniques in detail, it may be necessary once surgery. We might wish to obtain some idea of the methods that patients favour
more to stress the need to plan in advance what use will be made of them and in trying to cope with anxiety, in particular whether they find that talking to
the way the analysis is to take place. Will the results contradict other data? Will other patients is helpful or important. To begin with, we might use a fairly 'open'
they be used to interpret the results of more direct methods of questioning? At item, as follows:
what 'level' do we want to explore our respondents? What relative importance
do we intend to give to 'rational' and 'irrational' factors? Careful pilot work The best way to ouercome fear is. . . .
should make it possible to answer most of these questions; failure to plan in say a prayer
advance will often result in large quantities of unanalysable or contradictory don't think about it
information. ask for help from above
to forget it
SENTENCE
COMPLETION (see also Pilot Work, Chapter 4) Pray
to try to face up to it
As the name suggests, this device consists of a number of incomplete sentences get to know your doctor
- sometimes just a single word - which the subjects are asked to complete, not to think of it
usually in writing, with the first thing that comes to mind. In clinical psychology write down these fears and examine them
there are standard sets of sentence openings with standard scoring methods. to admit to it. No one is fearless
216 Quesfionnaire Design, Interviewing and Attitude Measurement Projective techniques in attitude study 217
Talking fo others about your coming operation. . . . Rather than have specific categories, such as 'dress', 'temperff 'visits',
I don't 'strangers', one might try a slightly more ambitious category such as 'social
you find things out defeats'. From the few cases we have here, it would seem that embarrassment
is not one of my pet subjects, better try and forget over 'social defeats' is more common among the upwardly mobile respondents.
doesn't worry me - I like to sympathize As with open-ended questions here, too, one may be led to a particular form
is very good of categorization by the need to explore a particular hypothesis, or by the data
helps one to feel happier about having them themselves - as in the above example. When developing such categories,
no use usually on a pilot sample, it is often helpful to divide the respondents into the
does not really appeal to me sub-groups of concern. We can then start building the coding frame by asking:
one can dispel fear in what ways do these groups of people differ in their responses to this item?
pointless, they cannot understand it Here is a further example in which the results are more readily and more
meaningfully classified after they have been subdivided according to the
Thus, each type of item has its function, and this has to be kept in mind when respondents' background. They were obtained in an inquiry carried out among
the sentence beginnings are designed. Pilot work is most helpful here, for some nurses in a psychiatric hospital. Observe the nature of the differences between
items yield quite unexpectedly varied results, whereas others produce very student nurses, and nurses with twelve or more years of experience, to the
stereotyped responses. After a time, one acquires a flair for writing good following item:
sentence beginnings.
How is this type of data to be quantified? With the more direct kind of item I wish that doctors . . .
the results can usually be coded or classified much as one would code the results STUDENT NURSES
of an open-ended question (see Chapter 14). The same approach, in principle, is would sometimes be more understanding
adopted with the subtler items, but a more 'interpretative' coding frame is often had more time to teach and discuss with staff
required. For instance, suppose that we had given the following item (among were a little more unbending sometimes
others) to three groups of married women - middle-class, working-class and were more considerate
socially mobile women: ' I Q I embarrassed and uncomfortable when . . .' Here are some had more time to discuss why they are doing certain kinds of treatment
of the results: would sometimes show more humour
NURSES WITH TWELVE OR MORE YEARS OF EXPERIENCE
WORKING-CLASS
would come to the ward on time
I am wrongly dressed would have patience with their patients
I have lost my temper were more ready to appreciate nursing experience
a visitor calls and I have an empty cupboard would occasionally discuss patients with staff
my child repeats something about somebody we have been discussing would find out from the nurses the condition of certain patients
my four-year-old daughter makes loud remarks about deformed persons would spend more time on the wards
MIDDLE-CLASS took more notice of nurses' knowledge
I can't hear properly would discuss more fully with staff
other people's children climb on my furniture
my husband introduces me to his female business associates There seems to be a general desire for more and better communication with
218 Questionnaire Design, Interviewing and Attitude Measurement Projective techniques in attitude study 219
the medical staff, but the tone of the younger nurses is more sad than angry, 'When I grow up I want to become an engineer,' said Mary, 'but it may be
more eager than cross, whereas the older nurses feel passed by and are evidently difficult for me because ..................................................................................................
critical of the doctors.
Perhaps these examples also illustrate what we mean when we say that
projective techniques can be revealing without the respondents being aware of
this and also permit greater depth and subtlety. 'When I grow up I want to become a nurse,' said John, 'but it may be difficult
Here are some rich, though unclassified, responses of university students to for me because ..................................................................................................................
the item:
If my father. . .
had more respect for others' intelligence, my mother would be happier
had received a better education, he might have got further than he has There were twelve items altogether, and these were mixed with a number of
had a quieter temperament he would be happier other items using occupations which in Western societies are usually held by
has so many hobbies, he isn't able to look after his family men and women more or less equally, so as to conceal the purpose of the
had not left home, things would have been easier questionnaire from the children. When piloting the items, some occupationshad
had loved me less, he would have disciplined me less to be replaced by others, but essentially the technique worked well and the
had not remarried a woman with a son of 19, life might have been less children readily 'projected' themselves into the person of the speaker in each
complicated item.
had been as much a friend as my mother, home would have been happier Obviously, the results produced qualitative information about the ways in
still which jobs are perceived by boys and girls still at school. It also proved possible
were not as he is, I would have disliked him to give each child an objective score indicating 'awareness of sexism' by counting
did not adore me so, I should not be so spoiled the frequency of responses such as 'because I'm a girl, and girls don't usually go
had been a rational person, life would have been easier. in for this kind of job' (whether the respondent was male or female). This shows
'
that, with a little ingenuity, we can make projective devices yield objective scores.
For items of this kind, the responses can usually be reliably classified as positive;
neutral or mixed feelings; negative; factualldescriptive. However for some
purposes, different and more insightful schemes could be devised, though these
might be less reliable. Here the respondent is presented with a picture or a drawing such as one might
find in comics or in cartoon strips, with 'balloons' indicating speech. However,
one of the balloons is empty and the respondents' task is to fill it. They will read
Objective scoring of projective materials and study the cartoon, then place themselves in the situation of the character
with the empty balloon and answer accordingly.
Figure 12.1 (overleaf) shows an example that was used in a study with
Here is an example of a sentence-completion device that was designed to find out adolescents. The drawing has deliberately been left rather vague and schematic
to what extent school children are aware of sexism and gender barriers in the to facilitate self-identification. As with the sentence-completion technique so
world of work. A dozen items were drafted, each using an occupation whch was here, too, the answers can be classified 'projectively', according to the attitudes,
predominantly associated either with men or with women. In each of the frames of reference and role perceptions that they reflect. In our example, the
'stories' the name of the speaker suggested a child's gender, and the names were results would show us a good deal about the respondents' attitude to authority,
so juxtaposed with the chosen occupations that gender barriers might be and there might well be social-class or other group differences in their reactions.
experienced. However, the wording of the items merely suggested that there There is almost no limit to the types of ~roblemsthat this technique can be
might be 'difficulties' and it was up to the respondent to indicate what these made to explore, and it is readily adaptable to all kinds of consumer research. It
difficulties might be. Three of the items follow: can and has been used in ordinary field interviews. A suspected disadvantage is
that the respondents may occasionally fail to identify with the character in the
WHEN I GROW UP
cartoon and therefore give an answer that tells us very little about their likely
'When I grow up I want to become a pilot,' said Anne, 'but it may be difficult reactions in a similar situation. Careful pilot work, to make sure that the pictures
for me because ................................................................................................................... are perceived as intended, is essential.
It is possible to apply this technique without the use of pictures or drawings.
The respondent is asked to imagine the situation from the words of one of the
220 Questionnaire Design, interviewing and Attitude Measurement Projective techniques in attitude study 221
some pictures or drawings will be created that relate to the problem at hand.
Figures 12.2 to 12.5 (on pages 222-3) are some pictures that were used by
Caudill (1958) to explore attitudes and social relationships in a psychiatric
hospital; the pictures were shown to both patients and staff.
In social research our aims are general, not personal or clinical. We will
probably want to gain an understanding of the attitudes of a large number of
people toward a particular problem or relationship. Therefore, we will probably
use fewer pictures, and they will be focused on our specific research problem.
Caudill, in his exploration of relationships inside a psychiatric hospital, used
pictures dealing with typical (though still ambiguous) hospital scenes. His results
did not enable him to do a character analysis of each of h s respondents, but he
learned a good deal about their attitudes and feelings from the way they
perceived and interpreted the pictures.
The technique has been widely used to measure ethnic prejudice, and
achievement motivation. A typical example of its use in the commercial field
might be some pictures showing a motorist at a garage or talking to a mechanic,
situations that could prompt various fears, frustrations, authority problems and
other attitudes. It has been shown, for instance, that some motorists regard the
mechanic as a friend; others treat him more like a servant; and to a third group
he is a mixture of doctor and magician.
As the purpose of the test becomes less broad and fundamental, so the type
of analysis will become more specific. We may no longer require our respondents
to tell a story but, instead, ask them to answer a number of questions about the
picture, or to rate it on a number of characteristics. Such questions or ratings
need to be so designed that they will seem quite reasonable to the respondents,
yet require them to interpret the picture and to use their imaginations, thereby
Figure 12.1. Teacher catches adolescent boy arriving late for class revealing their own attitudes. For instance, we may show a picture of a baby
with a rubber dummy in its mouth, followed by a series of questions including
some about the baby's mother (who is not shown in the picture). We may ask
respondents to guess her age, her social background, the number of other
characters (shopkeeper to lady customer, 'I'm sorry you are returning this children she has, and so on, in order to see whether the practice of giving a baby
vacuum cleaner, madam. Can you tell me what seems to be the matter?') and a dummy to suck is thought of as something that is done only in certain social
responds accordingly. Even multiple-choice answers may be provided. The classes, or only in large families where the mother is very busy, or only by young
cartoon technique is perhaps at its best when dealing with perceptions of human and inexperienced mothers. Such questions will need to be embedded in groups
relations such as authority, social class, queuing, games, buying and selling. of other questions, and a control sample should be shown the same picture but
It would be a mistake to assume that subjects would necessarily react to such without the dummy.
situations in real life in the way that their responses might suggest. The People are surprisingly willing to guess. In an investigation of attitudes
responses reflect attitudes and feelings, but their behavioural expression will towards types of alcoholic drinks (beer, whisky, wines, cocktails) one can show
depend on the actual situation. respondents pictures of many kinds of people, and guesses about their favourite
drinks will be readily forthcoming. A picture of a loaded carrier bag will produce
'character sketches' of the good and the bad shopper, the bargain hunter, the
PICTURE
INTERPRETATION buyer of convenience foods, the boutique addict, and others, according to the
This technique, too, has its origins in a clinical device, the well-known Thematic type of goods displayed. The need for control samples and control pictures must
Apperception Test. The test consists of twenty cards, one blank and nineteen be borne in mind, to find out which cues respondents are using to arrive at their
with various vague drawings on them. The drawings represent, for the most guesses.
part, human figures in situations that the respondent is asked to describe and to Pictorial techniques are particularly useful with young children. For instance,
interpret in the form of a story. a picture-inset test has been designed to test attitudes to skin colour among
In social research, one or more TAT cards may be borrowed, or more likely three-year-old children.
222 Questionnaire Design, Intemiezuing and Attitude Measurement Projective techniques in attitude study 223
Figure 12.4. The night nurse making rounds on the open ward
Figure 12.2. A patient at the front door of the hospital (either entering or leaving)
Figure 12.3. Evening on the open ward Figure 12.5. A patient in seclusion
224 Questionnaire Design, Interviewing and Attitude Measurement Projective techniques in attitude study 225
A ten-year-old boy, with no previous history of disturbed behaviour, is brought to you by his
mother because, since they moved fo the district, he has been difficult about school attendance.
Stories are most commonly used when we wish to test one or more specific He has taken to malingering and playing truant and seems generally 'nervy'. Physical
hypotheses. The respondents are presented with a brief account of some events, examination is negative.
and at the end they are asked to make a choice, say, between two characters in
the story, or to state which course of action they themselves would have I. I would not regard this as a medical problem.
adopted, or how the motivation of one character in the story can be explained. 2. I would treat the boy symptomatically (for example, with sedatives) but
Often, therefore, the story technique, though projective, resembles the 'closed' would make it clear that his school attendance was a matter for the school to
type of question. deal with.
For instance, here is a story that has been used to examine the attitudes of 3. I would regard this as a case for firm reassurance and positive advice (for
adolescent boys toward saving: example, I would counsel the mother on how she should handle the problem;
in addition, I might get in touch with the head of the boy's school to ensure
Tim uses his pocket money to go to the cinema, to sports events, to buy sweets, books and so proper management of the boy at school).
on. It gives him a lot of enjoyment now, but he hasn't got anything put away for the future. 4. I would spend some time with the boy and afterward his mother, trying to
Jack saves as much of his pocket money as he can. He will have money if he needs it in the bring out the underlying emotional difficulties; if indicated I would continue
future, but he goes without many pleasures now. with psychotherapy.
Which of the boys is using his money in a better way? 5. I would refer this patient to a child guidance clinic or psychiatrist.
1... .Tim
2 . . . .Jack
Sometimes the story technique can be combined with the use of incomplete
sentences. Here are some examples from a story that has been used successfully
Note the care taken in the wording of the question so as to make neither with adolescent boys, together with one set of responses:
alternative overwhelmingly more attractive.
For a more complex and subtle example, we may turn to a study of the THIS IS A STORY ABOUT A BOY CALLED JOHN
attitudes of doctors. General practitioners are sometimes asked to help with all
kinds of social problems, which may or may not be regarded as part of a doctor's 1. John came to a new school. He had hardly been there a few days when he already made
job. Some of these involve them in a semi-psychiatric role. To find out how friends because. . . he was the sort of boy who attracted everyone's attention
doctors feel about such problems, a sample of general practitioners was asked to for no reason.
respond to a set of case vignettes from which the following examples are taken: 2. Coming home from school one day, he happened to hear his father and mother talk about
him in the other room. His father said: . . . 'I wish John could do better at school.'
A married woman complains that she cannof go on looking after her widowed mother who 3. And his mother said . . 'But you know that he is not very good at learning.'
lives with her. The latter has a hemiplegia and obviously needs care but seems cheerful and 4. In the evening, after he was in bed and the lights were outl he was thinking of girls. He
fairly alert. The daughter however is tearful and agitated and insists that 'something must thought: . . . 'There's that horrible Sharon down the road always running
be done'. after me, can't she leave me alone?'
5. John had a friend he liked very much. Next day he said to his friend: 'I want to tell you
1. I would not regard this as a medical problem. something which you must not tell anybody. The secret is: . . . You know that
2. I would treat the patient symptomatically (ie, with sedatives andlor anti somebody put a drawing pin on Mike's chair, well, I did it.'
depressant drugs) but make it clear that I would not intervene in the social 6 . After tea, in the evening, fhefamily was sifting together. John was looking at his mother
situation. and said to himself: . . . I think she understands me more.
3. I would feel the most important thing here would be to give the patient some 7. John thought: I love my parents very much, but. . . it isn't my fault if I don't do so
relief from her burden. I might try to get a geriatric bed for the old lady for good at school.
a few weeks or alternatively arrange for a daily home-help. 8. John thought of his future. He said to himself: When I am a man I'll . . . be blowed
4. 1would spend some time with the patient trying to bring out the underlying if I'm going to do the job Dad wants me to do.
emotional difficulties. If indicated, I should continue with psychotherapy.
5. I would refer this patient to a psychiatrist. Analysis of the results can take place in two stages: for each item separately and
then, in a more thematic way, for the story as a whole.
226 Quesfionnaire Design, Inferviewing and Affifude Meamremenf Projective techniques in affif ude study 227
PSEUDO-FACTUAL
QUESTIONS (see p. 58) stereotypes, prejudices and feelings. In interpreting the data later on, we must
take into account the fact that the subjects have been 'forced' to some extent.
Under this heading comes a variety of techniques in which people are asked Apparently simple questions asking for some obvious explanation - ques-
seemingly innocuous questions of knowledge or belief which can be analysed tions so simple that they provoke respondents a little -can be most fruitful. In
projectively. A very simple example might be the question: 'What percentage of the previously mentioned inquiry into the attitudes of hospital staff, the nurses
the people in this town are foreigners?' Here, the true percentage is probably were asked to imagine that they were taking a visitor around the wards and had
quite small, yet there will be a wide variety of estimates, and the size of the to answer a number of questions: for instance, 'Why are uniforms worn?' As we
estimate has been found to correlate positively with anti-foreigner prejudice noted earlier, this is a rather ambiguous question, since it is not clear whether
scores on attitude scales and similar overt techniques. the question refers to the wearing of uniforms by doctors, patients, nurses, or
If people are asked to estimate the results of public-opinion polls, they will attendants and since it is not always clear whether, say, a suit of hospital clothes
tend to show a bias in favour of their own opinions. Similarly, estimates of the is a 'uniformJ. Most nurses referred to the uniforms worn by hospital staff, but
average pay of directors and workers, or of profit percentages, will reflect their attitudes were very different; some spoke of the practical side, keeping
political attitudes. The analysis of the results need take no account of the true clothes clean and safeguarding hygiene; some said it helped confused patients to
answer, if there is one; the distribution of the high and low scores within the know whom to approach; some said it was a mark of distinction and ensured
sample can be used. respect and authority; others thought the uniform was just 'traditional' and
The inclusion of one or two fictitious items among objects to be rated can be created a psychological barrier between themselves and the patients; still others
very successful. Suppose we are holding an 'immigration' survey asking said that a uniform helped the nurse to feel secure. When such responses were
respondents to say whether people from various other countries or of different classified, clear differences emerged in the attitudes of different grades of staff
races or religions should be allowed to come to this country, or should be kept -yet to the nurse-respondents the question seemed not attitudinal but factual.
out, or sometimes be kept out. We might insert among these the names of one
or two fictitious groups ('Danireans', for instance, has been used successfully). PLAYTECHNIQUES
When respondents come to such a name, referring to a group of people of whom
they have never heard, they will almost never admit this. They will feel that a With children, and even with adults on occasion, certain play techniques can be
response is expected of them, and they will proceed to engage in argument with used. In an interview, various objects are shown to the respondents to get them
themselves, along the following lines: 'Danireans? Hmm, never heard of them. talking; they may then be asked to do something with them; for instance, build
I suppose they are all right; I've not heard anything said against them. Let's have a miniature house with blocks and furniture, which may reveal their attitudes
some in and see what they are like'; or else 'Danireans? Hmm, never heard of via different spatial relationships (such as the proximity of parents' and
them. Who knows what they might be like? Better be safe than sorry, we'll keep children's bedrooms, or the size of their own room versus that of others). In the
them out.' Unwittingly, therefore, the respondents tell us what their attitude is psychiatric clinic, doll-play techniques are used both for diagnosis and for
and what kind of world they live in: a world in which other people are dangerous therapy.
and potential enemies until proven to be friendly; or a world in which most A useful device, which could easily be adapted to other purposes, is the
people are probably all right, unless there is some evidence to the contrary. Such Bene-Anthony Family Relations Test (Bene and Anthony, 1959). The child is
projective responses correlate well with other prejudice scores. asked to state who are the members of his or her family, and for each person a
The 'guess who' device is another seemingly factual approach that can be used cardboard cut-out figure and a 'post box' are provided. The child is then given (or
projectively (see p. 113).We may ask: 'These are the people who have too much has read aloud) a number of 'messages' such as 'I wish I could keep this person
power in our community. Who are they?' or 'These are the people you can never near me always,' or 'I want to do things just to annoy this person in the family,'
be sure of, you never know what they are thinking. Who are they?' And so on. and the task is to 'post' these messages. Provision is made for messages that
The items may be suitably varied to fit our own particular problem, and they apply to more than one person and for a 'Mr Nobody', who receives messages
may deal with the more stereotypical level or with deeper feelings. The illusion that apply to no-one in the family. The test can readily be scored to show
of a 'general knowledge test' can be kept up by having a few items at the affection, ambivalence and hostility to different family members, and can be
beginning and at the end that are innocuous. used with children up to the age of adolescence. Adults can be asked to complete
In many of these techniques the respondents are under some degree of subtle the test in accordance with the way they remember their childhood feelings.
pressure. The 'rational' answer to many of these questions would be 'I don't Getting respondents to sort objects or cards with labels on them and
know,' or 'the question is meaningless to me,' but since the question is part of a questioning them non-directively afterwards can be immensely reveahng.
survey, printed on a form, presented by a sane and competent interviewer, who People can be asked to sort or group almost anything: various kinds of pets,
apparently expects a simple answer to a simple question, the respondents feel children's misdemeanours, holiday resorts or chocolate bars; first, spontane-
that they must say something; and since there usually is no true, factual ously, in whatever way they like and in an unlimited number of groups and then,
knowledge on which they can base their answers, they delve into their attitudes, perhaps, with some direction such as 'now put together all those that would not
228 Questionnaire Design, Interviewing and Attitude Measurernenf
be suitable for young children'. This may be repeated several times, with
different directions. In this way we obtain, first of all, the respondents1frame of
reference, the way they see things: for instance, do they group holiday resorts
geographically or according to distance, according to the type of facilities they VARIETIES AND APPLICATIONS
offer, according to prestige associations and likely cost, or in terms of their
suitability for children? We also obtain some idea of the importance to them of
specific points such as the name of a product's manufacturer, the intention that
may lie behind a child's misbehaviour, or the small differences in price between
chocolate bars. Non-directive sorting tests can show how often such criteria are
used spontaneously as classificatory principles, and by what kinds of respond-
ents. Sometimes, this can lead to the development of multi-dimensional scaling
techniques.
Selected readings It is not always possible, within the constraints of time and resources, to apply
the full rigour of the linear-scaling model (see Chapters 9 & 11)to every
Campbell, Donald T., 'The indirect assessment of social attitudes', chapter 19 in Fishbein, technique we use in our survey. There are times when a 'quick and dirty'
M. (ed.),Readings in Attitude Theory and Measurement, Wiley, New York, pp. 163-79. approach is the only way open to us. However, if we bear in mind the basic
requirements of measurement, it soon becomes obvious that some methods are
'dirtier' than others. As a general guideline, when there is no opportunity for
SENTENCE
COMPLETION TECHNIQUES depth interviewing, detailed conceptualization, pre-testing, validation exercises
Hoeflin. Ruth and Kell, Leone, 1959, 'The Kell-Hoeflin Incomplete Sentences Blank: and so on, we should first and foremost strive for reliability. This means that it
Youth-parent relations', Monographs of the SocietyforResearch in Child Development, XXIV No. is better to have a group, or several groups, of items rather than a single item,
72. and that we should apply some form of item analysis to achieve internal consistency;
that is, reliability. If we can show that the items 'measure the same thing' then
it may be possible, later on, to attain some form of validity; or we may be able to
PICTORIAL
TECHNIQUES apply factor analysis to the data set to discover the underlying components of
Atkinson, John W. (ed.), 1958, Motives in Fantasy, Action and Society, Van Nostrand. New our instrument. There may not be time or opportunity to apply such statistical
York. techniques during the pilot work and pre-testing, and we may have to
A series of studies using pictures from the TAT to measure needs such as administer the entire group of items to all our respondents. In that case the
achievement, affiliation and power, complete with scoring systems. internal consistency analysis can take place later. So long as we have not tried to
measure attitudes, beliefs, awareness, knowledge or percepts by means of single
Caudill, W., 1958, The Psychiatric Hospital as a Small Society, Harvard University Press, items, we should be fairly safe.
Cambridge, Mass. Surveys have come a long way from the early days of biased samples and
crude techniques. Nowadays survey researchers are often asked to tackle subtle
PLAYTECHNIQUES and complex issues such as the determinants of voting for a political party,
people's images or stereotypes about another country, childreds perceptions of
Bene, Eva, and Anthony, E.J., 1959, The Family Relafionr Test, manual published by the social institutions such as trades unions, or people1sfeelings of control over their
National Foundation for Educational Research, Slough, Bucks.
own health (the health locus-of-control model). Survey researchers are thus
frequently required to exercise great ingenuity and creativity in designing new
types of techniques and adapting them to fieldwork conditions. In this chapter
we shall consider a variety of such alternative techniques, most of which are
elaborations of the 'closed' type of question; that is, they do not require the
respondent to write anything but only to tick the appropriate response
categories. Usually such techniques aim at generating some kind of score; they
are suitable for postal and group-administered questionnaires and can also be
used in standardized face-to-face interviews, perhaps with the help of show
cards for the respondent.
230 Questionnaire Design, Interviewing and Attitude Measuremenf Varieties and applications 231
Before going into detail it may be helpful to make a rough-and-ready meaningless. It assumes that we know which traits are important, that these
distinction between 'objective' and 'subjective' measures. When an interviewer is traits are unitary and not complex entities, that the trait names or labels mean
asked to assess the socioeconomic level of the respondent's dwelling area, the the same things to different people at different times, or to one person on
intention is to obtain an 'objective' rating of the neighbourhood (though, repeated occasions, and that the raters have the necessary information. This
inevitably, such a rating will also contain subjective elements, which we shall type of rating is of doubtful validity and suffers from gross unreliability. Here is
strive to keep to a minimum). If, however, we ask the respondent about the an example from a student project.
brand image of a particular soft drink, then we intend to use this information
'subjectively', to tell us something about the respondent rather than about the Example 13.1.
drink. In this case we are not interested in what is 'out there' but in what is
'inside'. However, this distinction between 'objective' and 'subjective' measures ASPECTS OF THE LIBRARY
is not always clear-cut. For example, when dealing with subjective responses we
shall want to collect these and add them up, to form collective images, profiles, Please give a rating from 0-10 (10= good) on the following:
representations, percepts or stereotypes. But, once we assmble a considerable Rating
number of subjective impressions they almost cease to be subjective and come to
represent some kind of 'objectified subjectivity'. This, in turn, can be related to 1. Provision of seats ...............................................................................
what is 'out there' and thus becomes a perceptual attribute. For example,
objective economic indicators may show that a certain country is 'rich', yet 2. Heating and ventilation ....................................................................
survey research may show that many potential holiday visitors subjectively 3. System of classification of books on the shelves ......................
perceive it as 'poor' and, since behaviour is more often determined by subjective
impressions than by 'objective facts', this has important implications for tourism. 4. Catalogue system ...............................................................................
5. Colour schemes and decoration ....................................................
6 . Quietness ..............................................................................................
7. Attitude of staff ..................................................................................
Rating gives a numerical value to some kind of assessment or judgment. Familiar
examples are school marks or grades, and proficiency reports on personnel in 8. Lighting ..................................................................................................
industry which attempt to give an objective assessment of an individual's 9. Hours of opening ................................................................................
performance. Usually, a set of assumptions is made about equality of the
intervals on a rating scale. 10. Table space ............................................................................................
We may apply ratings to almost anything - to individuals, to objects, to
abstractions and to ourselves. The task is usually to give 'so many marks out of Part of the problem here is with the various items or dimensions and their
so many', or some percentage equivalent. Sometimes, graphic rating scales are meaning: different respondents will probably have something very different in
used in which the response consists in making a mark on a line running between mind when they assess, say, 'attitude of staff'. Another problem concerns the
two extremes. use of numbers. Respondents have been provided with an arbitrary range (0-10)
The use of ratings invites the gravest dangers and possible errors, and in and told which is the favourable end (1.0 = good; often even this elementary
untutored hands the procedure is useless. Worse, it has a spurious air of precaution is omitted, so that some respondents use 'I' for 'best', while others
accuracy, which misleads the uninitiated into regarding the results as hard data. use 'IO'), but have otherwise been left to their own devices. Just what would be
For instance, hospital nurses are often rated by their ward supervisors on such the meaning of, say, a '4' on quietness? And would respondents use such ratings
attributes as practical ability, appearance, neatness, cleanliness, emotional consistently with themselves (across the items) and with each other? A third
stability, professional deportment, conscientiousness, reliability, punctuality, problem is somewhat subtler and is known as the 'halo effect'. Instead of
self-reliance, common sense, physical stamina and so on. The ratings may run assessing the library on its merits regarding each item separately, respondents
from 1-5, or from 1-10,or any other intervals. These ratings are taken into let themselves be influenced by their general feelings of like or dislike.
account when matters arise concerning promotion and transfer or the provision A different form of the halo effect expresses itself in a response set. If the
of references. Yet in a typical investigation, if the same supervisor rates the same rating scales are arranged one underneath the other and always with the 'good'
nurses on two occasions, the correlations between the first and second rating (socially desirable) end on the left-hand side and the 'bad' end on the right, then
will be very low and will only occasionally reach statistical significance. The the respondents - having once made up their minds that they are favourably
correlations with assessment of these nurses by other supervisors, and with disposed toward the object of the ratings - may run down the page always
examination results, will be similarly low. The procedure here is, in fact, checking the position on the left, or vice versa, without actually reading the
232 Questionnaire Design, Interviewing and Attitude Measurement Varieties and applications 233
items or giving each of them separate thought. To counteract this, we try to a neutral category or midpoint, to one side of which lie the favourable categories
randomize the direction of the scales, so that the socially most desirable end falls and to the other side the unfavourable ones. This is also because raters are a little
sometimes on the left and sometimes on the right. afraid of using the extreme categories - a phenomenon known as the error of
Having the rating scales on separate pages further helps to offset the halo central tendency. We can try to make our extreme categories sound less extreme to
effect and is also useful in combating the logical error, the tendency to give an encourage their use, or we may reconcile ourselves to the inevitable and decide
object or a person similar ratings on two attributes that are thought to be afterwards to combine, say, the two or three extreme categories at each end -
correlated, such as intelligence and emotional stability, or price and quality. We in which case something like a ten-point scale would seem desirable. Although
must also make sure that our raters have the necessary information: we should steps on a rating scale are often used as if they had numerical values and as if the
not ask them to rate Greek grammar or the work of an electroencephalographer intervals were equal, we usually lack the evidence to justify such assumptions.
if they have no relevant knowledge. After deciding the number of steps in our scale, we have to define the
Perhaps the chief danger of ratings lies in the ease with which they can be extremes, or better still, define each step. If we have a seven-point scale of
influenced, often by variables of which the rater is unaware. For instance, 'punctuality' then we must say when, or by what criteria, a person is to be rated
suppose we asked a schoolboy's mother and his teacher to rate him on various five, or one, or three and so on. Is a person who is frequently late by a few
attributes such as irritability, shyness, obedience and so forth. We would minutes to get the same rating as someone who is never late, but who has been
probably find differences between the two sets of ratings, not only because the very late on a single occasion? Is earliness rated the same way as lateness? On
child may behave differently at school and at home, but also because the teacher
compares him with other children of the same age and sex, whereas his mother Example 13.2.
probably compares him with the others in the family and with a few children in
the neighbourhood. In other words, the two raters have different frames of Fels Parent-Behaoiour Rating Scale No. 3.15
reference. This is a problem that frequently affects our perceptions and Democracy of Regulation and Enforcement Policy (Democratic-Dictatorial)
evaluations and can lead to much misunderstanding. It is most important, Rate the parent's tendency to share with the child the formulation of regulations
therefore, when we ask for a set of ratings, to indicate the frame of reference for the child's conduct. Does the parent give the child a voice in determining
that the raters are expected to use. The label or title of the set of ratings will have what the policy shall be? Or does the parent hand down the established policy
an influence, too; in the preceding example, a title such as 'neuroticism scale' from above?
would probably yield quite different ratings from a title like 'Child Development Disregard immediate issues not covered by policy. Rate independent of
Scale'. It should be recognized that the respondents are also influenced by what justification of policy to child, and independent of restrictiveness of regulations.
they think is the purpose of the rating procedure. If we do not tell them what the Include both overt consulting with child and considering child's expressed
purpose is (by offering a title or a frame of reference), then they will try to guess wishes. Dictatorial policies may be wise or foolish, benevolent or selfish.
what the purpose might be, and since different respondents may guess very
differently, their ratings may diverge more than they might have done.
Ratings can be used in various ways: (1)as objective assessments - for Endures much inconvenience and some risk to child's welfare in giving child
instance, a rating of the quality of fixtures and furnishings in a home during or large share in policy forming. Consults with child in formulating policies
after an interview; (2) in a subjective, projective way to tell us something about whenever possible.
the rater's percepts and attitudes; (3) as self-ratings of personality traits or Attempts to adjust policies to child's wishes wherever practicable. Often
attitudes. Each of these types of rating scales requires a somewhat distinct consults child.
interpretation, and different aspects have to be stressed in their design and
quantification. Deliberately democratic in certain safe or trivial matters, but dictates when
In constructing an objective rating scale, perhaps the first step is to define the there is a sharp confict between child's wishes and other essential requirements.
dimension that is being rated. It should be possible to describe it so that it will Neither democratic nor dictatorial, deliberately. Follows most practical or
have approximately the same meaning to all the raters involved in the easiest course in most cases.
investigation. This may be relatively easy when a food product is being rated on
'sweetness', but it becomes more difficult when, say, we want to rate a family on Tends to be rather dictatorial, but usually gives benevolent consideration to
the 'extent to which the family supports the child in conflicts with the outside child's desires. Seldom consults child.
world'. Dictatorial in most matters, but accedes to child's wishes occasionally when
The number of steps in a rating scale tends to vary from four or five to they do not conflict with own convenience or standards.
perhaps ten. Raters are unable, in most instances, to make discriminations that
are finer than ten points or so. Many research workers favour an uneven Dictates policies without regard to child's wishes. Never consults child when
number of steps, especially when there are only three or five, in order to have setting up regulations.
234 Questionnaire Design, Interviewing and Attitude Measurement Varieties and applications 235
I
I
1
I
I
I
I
I
II
I
I
II
I
I
1I
I
I
I
Coldhearted
Takes good care of
her children
Selfish
1 Of very
great
importance
Of
great
importance
Of
some
importance
/ no
importance
1
1. Absence of
Patient I I I I I I I I Impatient
parents, family
Competent mother I I I I I I I 1 Incompetent
mother 2. Absence of close
friends
Lively I Quiet
I I I I I I I
3. Se~arationfrom I I I I I
Very like myself r I I I I I I I Not at alllike wGe, husband,
myself children
Scatterbrained I I I I I I I I Sensible 4. Missing your
own language,
Self-assured I I I I I I I 4 Always worried
books, magazines
Unhappy by nature I I I I I I I I Happyby nature etc.
Bad at housekeeping I I I I I I I I Good at 5. Missing the
housekeeping festivals,
Takes good care of her Pampers her celebrations,
children social functions
children I I I I I I I I
etc.
Would like to have as a
friend
Would not like to
have as a friend 6. Food difficulties
7. Discomforts of
c
working in a
what occasions is punctuality to be assessed? Over what period? These and household
many other questions have to be given unambiguous answers, and we have to 8. Long period of
give all raters full and clear instructions. Unless we do this, two raters rating the absence from
same person or object will have conflicting results. home
By way of illustration, an excerpt from the Fels scales for the assessment of
child-rearing by means of home visits is reproduced in Example 13.2. Note that 9. Boredom and
each step, as well as the general dimension, has been described in some detail. monotony of
When objective assessments are wanted, it is dangerous to give such labels as present life
'frequently', 'moderately', 'often', 'occasionally' and so on to the different grades, 10. General sense of
since they can be interpreted differently by the various raters. psychological
We often use rating scales when an attitude or a personal impression is (emotional)
wanted in subjective rather than objective terms. For instance, suppose we wished depression
to employ an actress in a half-minute advertising film concerning baby foods.
Our choice may have been narrowed down to three or four candidates whose 11. Other reasons
photographs (with babies on their laps) may be shown to an audience of (please specify)
mothers. To find out how the audience perceives these 'mothers' we might ask
236 Quesfionnaire Design, Inferuiewing and AffitudeMeasurement Varieties and applicafions 237
a group of women to fill out the graphic rating scales that are shown in Example As we saw above, there are risks in using rating scales; for instance, are we
13.3. justified in basing our calculations on the assumption of equality of intervals,
Here, where only a rough, personal impression is required and where the both within each scale and between different scales? Osgood offers valuable
differences between the candidates are more important than the absolute evidence on the validity, reliability and sensitivity of his scales and, in any case,
figures, only the extremes of each scale have been indicated.Note, however, that some of the most interesting types of analysis permitted by this technique refer
the 'favourable' extreme is sometimes on the right and sometimes on the left, to not to assessments by groups of respondents, but to the subjective semantic
try to counteract response set and halo effect. The scales are deliberately worlds of single individuals.
contrived to encourage guessing, to obtain attitudes and first impressions. We If the semantic differential is simply regarded as a set of rating scales, it can be
must be very clear in our own minds that we are not getting an 'objective1 used to obtain the percepts of various political personages, .different national or
assessment but rather a snap judgment and sometimes a wild guess - for our ethnic groups or any other subject matter. If we are willing to average sets of
present purpose, that is what is required. ratings given by groups of respondents, then we can compare the results
Sometimes, rating scales are used to tell us something about the rater rather produced by differentgroups for the same concept or object. Thus, we can show,
than about the object of the ratings. The work of Osgood (see below) on the for example, that women, on average, regard a given brand of cigarettes as less
semantic differential is important, particularly in the clinical field. For instance, mild (stronger) than men do.
suppose we take again the set of mother-rating scales discussed above and ask Example 13.5 illustrates the use of the semantic differential technique. It
a woman to rate herself, her own mother and also the 'ideal' mother on each of shows a set of rating scales developed for research with players in international
the scales. From the results, we might be able to see how she perceives her own crisis simulation games, to obtain their percepts of different countries. Previous
mother, how far she looks upon herself as similar to her mother, to what extent pilot work on a much larger number of scales had shown, with the aid of factor
her image of her mother and the 'ideal mother' overlap and where she feels that analysis, that these percepts were determined by five underlying factors. In the
she herself falls.short of the ideal. For such an investigation we would no doubt final instrument each of these was defined by six semantic differential scales,
wish to add other variables and change some of the existing ones (see Selected making thirty rating scales in all.
Reahgs). The following factors were obtained:
Example 13.4 gives a simple set of self-assessment ratings from an inquiry by
Postlethwaite (1962) into the problems of continental students living in Great
Britain.
1 Factor I: General 'niceness'; liking;
Factor 11: WesternlOriental or developedlunderdeveloped;
It is really a scale of subjective importance applied to each item in turn, and Factor 111: Toughlgentle;
each respondent supplies his or her own frame of reference. Such scales of Factor IV: Volatilelequable;
subjective importance can be applied to many kinds of persons, objects and Factor V: Crudelsophisticated.
hypothetical or actual situations. They yield attitudes rather than objective
assessments, and a crude total 'unhappiness score' could be calculated. With In this way we can obtain, for each respondent, five reliable subjective scores
more items, balanced and grouped into different areas, the technique may be expressing his or her perception of a given country, or of several countries, and
made to yield sub-scores and profiles, provided that internal consistency can be the percepts of groups of respondents may be compared country by country.
demonstrated. A factor analysis might be attempted, if suitable computing Note that the S-D scales deal not only with 'factual' aspects, such as 'Western1
facilities are available. Oriental' or 'secularlreligious' (which often are very much a matter of opinion,
anyway) but also with images and feelings, for example 'warmlcold', 'hardlsoft',
'masculinelfeminine'.
The semantic differential and the repertory-grid techniques The same approach can be used to obtain brand images of products, or the
public images of well-known firms or personalities. Each time extensive pilot
The semantic-differential technique was originally developed by Charles E. work, pre-testing and factor analysis must precede the choice of the final set of
Osgood (Osgood et al., 1957) and his colleagues as part of their quantitative scales.
study of meaning. It consists essentially of a number of seven-point rating scales Research has shown that S-D scales with seven intervals are usually optimal.
that are bipolar, with each extreme defined by an adjective: examples might be However, some investigators prefer to use five-point or three-point scales for
wiselfoolish, stronglweak, excitablelcalm. The respondent is given a set of such particular purposes.
scales and asked to rate each of a number of objects or concepts on every scale The technique really boils down to a selection of rating scales made up for the
in turn. It is possible to submit sets of such ratings to factor analysis, in a search particular purpose at hand, on the basis of pilot interviews. In setting up these
for the basic dimensions of meaning. Osgood has shown the importance of three scales, the location of the positive end should be randomized, so as to try to
factors (evaluation, potency and activity), though other factors (such as counteract any halo effect (see p. 231). If several concepts have to be rated, it is
tautness, novelty and receptivity) may also play a part. best to have a separate rating sheet for each one; thus, the same set of scales can
238 Questionnaire Design, Interviewing and Attitude Measurement Varieties and applications 239
Example 13.5. About: be given over and over again, each time with a different concept at the top of the
(country name) page. Care should be taken that the two descriptors at the extremes really are
opposed and do define some kind of scale or dimension between them. Some
Kind :: : : : : : : Cruel extremes have more than one opposite (for instance, sweetlbitter or sweet1
Trustworthy : -: -: -: -: -: -: -: Untrustworthy sour), while others have none (for instance, burning), so that instead of having
Malevolent : -: -: -: -:-: -: -: Full of good will
two extremes we really have one extreme and a neutral end (in terms of our
example, burninglnot burning). It is possible and often useful to obtain
Basically Basically
responses to rating scales that seem inappropriate to the concept under
peace-minded : -: -:-: -:-: -: -: war-minded
consideration, such as masculinelfeminine(with respect to a brand of cigarettes),
Break bargains : -: -: -: -: -: -: -: Fulfil bargains
or roughlsmooth (with respect to Socialism). By their more imaginative
Cold :: : : : : : : Warm approach, such scales can be used to cover aspects that respondents can hardly
Backward : -: -: -: -: -: -: -: Advanced put into words, though they do reflect an attitude or feeling.
Western : -: : -: : -: -: : Oriental The value of the technique depends largely on a suitable choice of concepts
Rich :: : : : : : : Poor and rating scales. Since the scales will be applied to all the concepts, the problem
of relevance or applicability must be carefully considered: can we, for instaqce,
Autocratic : -: -: -: -: -: -: -: Democratic
apply calmlexcitable to certain foods? Another issue is the scales' factorial
Influenced Not much composition, which is often not available. We are less restricted in our choice of
by Western influenced by objects, which may well be largely determined by the nature of the investigation
powers : -: : : -: : : -: Western powers
However, only pilot work kill show whether more imaginative objects, such a,
Ifiterate : -: -: -: -: : : : Literate 'my ideal self or 'the good housewife' can fruitfully be used.
Hard : :: : Soft As part of his broader theoretical framework, Kelly has proposed the
Weak : -: -: : -: -: -: : Strong repertory-grid technique, which has been taken further by Bannister and others
Hesitant : -: : : -: : -: : Decisive in several investigations (see Selected Readings).The technique is essentially an
Active : -: -: -: -: -: -: : Passive
individual approach, though it can be adapted for group sessions. Its purpose is
to supply a 'map' of the respondent's 'personal constructs' and their interrela-
17. Masculine : -: -: -: -: -: -:-: Feminine
tionships or changes over time. It is an attempt to find out how people 'see' their
18. Cowardly : -: -: -: -: -: -: -: Brave world, or at least that part of their world with which the investigation is
19. Feels secure : -: -: -: -: -: -: -: Feels insecure concerned, such as their family members, certain objects or abstractions, brand
20. Holds a grudge : -: -: -: -: -: -: -: Forget grudges names, other nations or groups, consumer durables and so forth. Repertory-grid
21. Shortsighted Longsighted testing is not a single test but a highly flexible technique which can be evolved
policies : -: -: -: -: -: -: -: policies afresh for each new investigation.
To begin with, it is probably best to let a respondent, say, a thirty-five-year-
22. Greatly Not much
concerned with concerned with old male, supply his own constructs. In accordance with Kelly's theoretical
prestige and prestige and approach, a construct is a way in which two things are alike and in the same way different from
saving face : -: -: -: -: -: -: -: saving face a third. We therefore start by getting the respondent to look at sets of three
23. Quick to escalate : -: -: -: -: -: -: -: Slow to escalate
objects, photographs, persons and so on and ask him to say in what way two of
these are alike, and in that way different from a third. For instance, the
24. Deliberate : -: -: -: -: -: -: -: Impulsive
respondent may be shown pictures of three men, and he may say that two of
25.
. . : -: -: -: -: -: -: -:
Rig~d Flexible them are 'Jewish-looking', while the third is not. Thus we find out what are, to
26. Guided by Guided by him, meaningful attributes of the stimulus material; these are his 'constructs'.
domestic foreign Such constructs are incorporated in a grid, down the left-hand side. Across the
considerations : : : : -: : : -: considerations top will appear the 'objects', the things, persons, photographs, nationalities, or
27. Looks to the Looks to the other stimulus material, to each of which the construct is to be applied, now that
past : : : : -: -: -: -: future it has been produced. Each construct is regarded, not as a matter of degree, but
28. Secular : -: -: -: -:-: -: -: Religious as a dichotomy: as an attribute, it is either present or absent. In the above
29. Strong leadership : -: -: -: -: -: -: -: Weak leadership example, the respondent may now be shown a series of photographs of men, and
30. Spends heavily on Spends little on in each case he has to say whether or not the person is 'Jewish-looking'. The
arms : : : : : :: : arms
investigator takes the respondent through the grid step by step, in each case
240 Questionnaire Design, Interviewing and Attitude Measuremenf Varieties and applications 241
placing a tick underneath each object that is said to possess the construct in technique, but for studies of change, in particular, it would well repay a detailed
question. (See Example 13.6 for an example of this.) Inspection now allows us to consideration.
make certain inferences; for example, two constructs that go together (both An especially valuable use of the repertory-grid technique is in conjunction
being ticked, or left unticked, all along the two rows) have similar or subsumed with the development of a set of semantic-differentia1scales. In a sense, the
meanings to our respondent; in the example below, 'intelligent' and 'Jewish- concepts can be used as pilot work for a set of S-D scales. For example, in
looking' go together in this way. Similarly, we may infer that two objects, in research on the ways in which people choose a new dwelling when they move
whose columns the patterns of ticks and blanks are closely matched, are home, the successive deployment of both techniques can show us how people
perceived as similar by the respondent. perceive their own homes, how they initially conceptualize the home they are
The most important aspects of the repertory-grid technique are the looking for, how they prioritize various attributes, and how these percepts and
constructs (attributes) and the objects (cards, persons, photographs) to be priorities gradually change after viewing a succession of dwellings. Both
included. These depend entirely on the purpose of the investigation and should techniques have also found favour among social geographers (see Selected
flow directly from it. Such purposes can, in turn, be classified as being concerned Readings).
with objects (to find out which person, things, photographs and so on are seen as
similar or in what ways objects are said to differ), with structure (to find out how
the meaning of different constructs hangs together, to study the individual Inventories and checklists
respondent's 'construct map') or with change (for instance, as a result of
psychotherapy or advertising).Variations in the technique are often introduced. An inventory is essentially a list that respondents are asked to mark or tick about
We may, for instance, ask the respondent to produce his own objects such as 'my themselves. It may consist of a list of interests, and the task may be to 'tick those
best friend,' 'the most intelligent person I know,' 'My favourite uncle.' Or we things that interest you a lot'. It may be a list of small social misdemeanours, and
may offer certain constructs to the respondent, rather than letting him suggest the respondent is asked to tick various columns to say 'how bad' each is. It may
them. This latter procedure has the advantage that it turns the repertory grid be a list of personality traits or emotional feelings, and people are asked to tick
1
into a group technique, enabling many respondents to fill in the same grid, and which of these apply to them. It may be a list of spare-time activities, in which
factor analysis as well as other statistical techniques can be applied. one has to tick the activity engaged in most often, and so on.
The repertory-grid technique is extremely flexible, allowing studies at the Inventories resemble subjective rating scales, except that they are designed to
personal level (for instance, of schizophrenic thought) as well as at the group yield two or more scores. The items - though commonly presented in random
level. It has been employed in such diverse inquiries as the study of social-class order - can be grouped in specified ways and scores can be computed for each
differences in language and thinking habits, group differences in value systems respondent on a number of such groups of items, or on the inventory as a whole.
and the perception of different kinds of houses. The assumption underlying each In the better types of inventory, in particular those that can be properly
variant of the technique is that we can usefully infer the way a person 'sees' the described as personality tests, the items are selected after careful pilot work, and
world from patterns of associations between constructs and objects. Kelly's the grouping into areas is done on a statistical basis, by means of factor analysis,
theoretical framework is not strictly necessary in order to make use of the so that those items that are scored together really belong together. However,
since there are always problems for which no inventory has been devised, many
people construct inventories and group items together largely on an a priori basis.
Example 13.6. In t h s way, they obtain a quick, relatively crude but useful set of measures, with
reasonable reliability because of the use of area scores rather than single
questions.
Photo A Photo B Photo C Photo D Photo E Here is one set of items, from an inventory dealing with the (perceived)effects
of television. The items refer to 'stimulation of activities':
Constructs
(a) I have copied the way people dress on television.
Intelligent - J J J (b) I have tried to cook something after it has been shown on television.
Tough J - J - - (c) I have made things after they have been shown on television.
(d) I have tried to read a book after it has been talked about on television.
Jewish-looking - J - J J (e) I have gone to a museum or art gallery after seeing it on television.
Likable - J - J -
The items are ticked as 'true' or 'untrue', and in the actual inventory they were
Selfish J - J - -
mixed in with many other items. Note that the item referring to cooking may
inflate the total area score for women; depending on the purpose of the
242 Questionnaire Design, Interviewing and Attitude Measurement Varieties and applications 243
Almost everyone at one time or another is worried about some things. Different
people are worried about different things. Read through the list and for each HOW BAD IS IT? HOW GOOD IS IT?
thing tick to show whether it worries you a lot, a little or does not worry you. How good or bad would it be, in your opinion, if a psycluatric nurse like yourself
on your present ward did any of the following things? (Please put a tick in the
Tick what is right for you. right column.)
1 1 1 1 1
Worries Worries Hardly ever
me a lot me a little worries me
1 21
Would
Not getting along well with other How bad is it if you not Fairly Very w k l d
children How good is it if you Bad matter good good depend
Not being popular flSpend a good deal of time
Feeling that some children would not talking to patients?
like to come to my home Let some patients remain
Feeling left out of things buntidy?
Other children making fun of me O Sometimes show that you
are in a bad mood yourself?
Being called names
Appear to the patients to
Feeling that some children look down be very busy and efficient?
Forget to put on a clean
uniform?
inventory, the relative balance of the items in each area should be kept in mind. Avoid discussion of
Inventories have a wide variety of uses; they can be constructed with various personal problems of a
purposes in mind or to test particular hypotheses. Taboo items, inadmissible patient with him because
problems or antisocial forms of behaviour can readily be inserted; respondents you feel that the doctor
who see such items in print often respond to them more readily than to an open should do this?
question. Example 13.7 is a set of seven items from an inventory of 'worries'
Take special pride in having
used with schoolchildren. This particular area covers feelings of rejection by patients who are quiet and
other children; it could be used, for instance, to explore the feelings of
do as they are told?
youngsters from a minority group going to an ethnically mixed school.
The scoring is 3-2-1, or 2-1-0, or some other arbitrary scores, and the items Talk about a patient when
are given equal weight. Note the careful wording of the instructions. the patient is present,
Inventories can sometimes be made to perform at a fairly subtle level. In a acting as though the
study of the attitudes of psychiatric nurses, a successful attempt was made to patient was not there?
discriminate between those with rather autocratic and repressive tendencies,
Get deeply involved with
and those with a more psychotherapeutic orientation. Example 13.8 is part of the
what happens to particular
inventory that was used. Note the permissive way in which some of the items
patients?
have been worded, ('Find that you have to be firm . . .' instead of 'be firm
with . . .') Also note the category 'it would depend', which was hardly ever used Find that you have to be
by the more autocratic kind of nurse. However, psychiatrists, who were also very firm with some
given this inventory to fill in, made frequent use of 'it would depend', partly patients?
because they were less aware of the nursing 'ethosJ.
It is possible to use factor analysis to determine the underlying components of
244 Questionnaire Design, Interviewing and Attitude Measurement Varieties and applications 245
Example 13.9.
A GOOD CITIZEN:
Imagine that you had to explain what a good citizen is or what a good citizen 24. Knows that hislher country may sometimes
ought to do. Please read each sentence, then put a tick ( J )under the heading make mistakes
'Good Citizen' if that is what you mean by a good citizen. If the sentence does
NOT help to explain what you mean by a good citizen, put a tick under 'Other'. 25. Gives presents to the police in the hope of get-
If you are not sure, put a tick under the question-mark ('?'). ting favours in return
26. Stands up when the National Anthem is
A GOOD CITIZEN: vl
Citizen
played
27. Shows respect for a funeral
*
I. Obeys the law
29. Tries to make the government change its mind
30. Demands hislher rights if they are threatened
-kl4-4
31. Knows a good deal about art and music
32. Helps to keep the streets and parks tidy
1
I I
a checklist or inventory, so that internally consistent scales or sub-scales can be
obtained.
Example 13.9 shows a checklist concerned with the description or attributes
of a 'good citizen' in a number of countries. There are three simple answer
categories and thirty-two items, culled from pilot interviews in different
9. Is good at sports countries; each item has been mentioned by at least some respondents as being an
important attribute of a good citizen.
Factor analysis has consistently shown that the inventory has an underlying
11. Works hard structure comprising three 'factors' or 'dimensions'. Each factor describes one of
three distinct types of good citizen, characterized by different sets of items: the
12. Joins a political party
active good citizen (who joins a political party, always votes, and takes an interest
13. Knows a good deal about how our tax money in public affairs); the passive good citizen (who pays taxes, obeys the law and is
is spent loyal to hislher country); and the disengaged good citizen (who expresses
citizenship in non-political forms such as hard work, respect for a funeral, and
14. Has good table manners
1 family loyalty). A score for an individual's perceptions of each of the three types
15. Studies hard to pass an examination
16. Pays hislher taxes regularly
s of citizen can be obtained by summing the scores on the items that comprise it.
Thus a respondent who endorsed all the items relating to 'the active good citizen'
would see a good citizen as active in a large number of ways, while a respondent
17. Minds hislher own business who did not endorse any of them, would reject the notion that 'the good citizen'
is 'active' altogether. After piloting and factoring, the inventory can therefore be
18. Keeps up with what is happening in the world
shortened to contain just sufficient items to identify the different percepts of a
19. Tries to change things in the government good citizen and yields three scores per subject, of adequate reliability. The items
have been so worded that they can and have been used with.adults, but also with
20. Gets other people to vote in elections children from the age of ten years upwards. The inventory can be included in
21. Is liked by most people any research project to which it might be relevant and which uses self-
22. Knows hislher rights I administered questions. Translations (which have themselves also been inde-
pendently factor analysed) are available in half-a-dozen languages (see Torney,
23. Is willing to serve on a jury f l l Oppenheim and Farnen, 1975).
246 Questionnaire Design, interuiewing and Attitude Measurement Variefies and applications 247
11 /I
12 /I
By sharing needles/syringeslblades with a person who has
AIDSIAIDS virus
By shaking hands with a person with AIDSIAIDS virus
I 1. Walking under a ladder brings bad luck.
2. A black cat crossing your path brings
3. You have to be more cautious on Friday the 13th.
13 7
1By having sex with many people 4. Seeing only one magpie brings bad luck.
5. It is bad luck to leave your shoeslslippers overturned.
14 T I By having sex with a man who has AIDSIAIDS virus
6. Putting shoes on a table means a snake will come into the house.
15 1 1 By having sex with a woman who has AIDSIAIDS virus 7. Opening an umbrella indoors brings bad luck.
16 I I I By using public toilets 8. Handing over a knife to someone (ie hand to hand) means that
17 1By being bitten by a mosquito or other blood-sucking insect you are going to have a fight with that person.
9. Throwing away spilled salt brings bad luck.
18 L_JJ By donating blood to a person who has AIDSlAIDS virus
10. A rabbit's foot brings good luck.
19 I( By blood transfusion/receiving blood from a person who has
I I . It is bad luck to cut your own hair.
AIDSIAIDS virus
12. It is bad luck to own peacock feathers.
20 1
7By having sex with a person who has AIDSIAIDS virus 13. You should avoid walking on cracks in the pavement to avoid
21 1Other (specify) bad luck.
248 Questionnaire Design, Interviewing and Attitude Measurement Varieties and applications 249
Example 13.12. which one can quickly collect information without having to ask a great many
questions, and the data can be analysed in several different ways. Particular
Will grown-ups generally agree about what our government should do, or do hypotheses can be explored in a way that does not make them obvious to the
they sometimes disagree? Below you will find different groups of people in each respondent.
question: please tell us how well you think they agree with each other about what The over-all frequency tabulation will immediately show which is the
the government should do, by putting a tick ( 4 )in the right column. At the top preferred method of treatment for, say, a dry throat. It will also show under
of the columns you will find the following headings: what circumstances a particular remedy, such as a syrup, will be applied. Next,
the maladies can the grouped - comparing, for instance, all the types of coughs
Mostly agree
Agree about half of the time ABOUT WHAT THE GOVERNMENT Example 13.13.
Disagree most of the time SHOULD DO
I don't know
1 1 Agree 1 1 1
Disagree
Mostly about half most of I don't
agree of the time the time know
Here are some well-known remedies that can be used to give relief with
different types of complaints and illnesses. For each of these illnesses or
complaints, please tick those remedies that you think are best:
with all the various throat disorders. Or we may wish to make a detailed intervals between classes or ranks -ranking is less precise. The commonly used
comparison of the profile shown by throat pastilles with that shown by throat prestige rankings of occupations are of this kind.
tablets. We can group respondents by age, or sex or in some other way to show Mention should be made of paired comparisons. Here, the objects to be ranked
whether older people are more inclined toward the 'traditional' remedies, or are presented two at a time, and the respondent has to choose between them. To
whether there is a sex difference in preference for chest rubs, and so on. Quite obtain a rank order from this procedure, all possible combinations of pairs have
likely the grid has some items in it that are aimed at particular hypotheses; for , to be presented, preferably in random order. For ten objects this requires forty-
instance, will harsher remedies be advocated for a smoker's cough because it is five paired comparisons (%N(N-I)), and the number of comparisons rises very
'your own fault'? steeply when the number of objects is increased. For this reason, paired
comparisons are not used much in social research; the method comes more into
its own in the laboratory (see Chapters 9 and 11).
Ranking (see also Chapter 9) Ranking can be used projectively in a useful and revealing way to tell us
something about the respondent rather than about the objects being ranked.
Ranking means arranging in order, with regard to some common aspect. We can Teachers, for instance, may reveal a social-class bias when ranking pupils on
rank children in terms of their school performance, soldiers in terms of their 'popularityJ.Nurses may identify themselves so closely with doctors that they
leadership potential, lettuces in terms of freshness, or paintings in order of tend to up-grade the prestige of their own job as compared to the way nursing
merit. There are practical limitations to the number of ranlungs most people can is ranked by the general population.
be expected to carry out; under normal survey conditions, to put ten objects in
rank order is probably as much as can be asked. Ranking tells us nothing about
the differences between ranks; the 'distance' between ranks two and three may
be very large, but between five and six it may be minute - ranking tells us the Example 13.14.
order or sequence, but the size of the rank intervals is unknown and unlikely to
be equal. This has led to the development of Spearman's Rho and Kendall's Tau What are, in your views, the most important causes of poor living conditions in
(the special correlation co-efficients for two rank orders) and to numerous other under-developed countries? (Please tick not more than three, and number them,
statistical devices, none of which requires any knowledge of the size of the rank I, 2 , 3 in order of importance.)
intervals. Indeed, we would not use the ranking technique if such knowledge
were available (or if suitable assumptions could be made about equality of scale The results of colonialism
intervals); we would use some form of rating or scaling, instead.
Before setting a ranking task we must be very clear about the results needed. The people not trying hard enough
For instance, to rank paintings in order of 'merit' may produce very confusing
findings because merit is a very vague term compounded of a liking for the I Freak weather patterns
An alternative to asking respondents to rank a substantial number of items or with regard to validity and selective recall and tend to yield underestimates. Daily
responses is to ask for a partial ranking, that is for a ranking of the three or four checklists, on the other hand, may yield ouerestimates; here the respondent is
preferred responses (see Example 13.14). daily given or sent a list of magazines, radio programmes, social activities or
Note that an 'Other (please specify)' category is almost always needed and that other relevant materials, with a request to fill in a list for that day. The diary
a 'Don't know' category should have been added. Observe also the carefully technique, used with proper safeguards, can yield results that are probably
worded answering instructions, which will make the statistical analysis more nearest the 'true' frequency for a given period, but a good deal of comparative
precise. research is needed to make sure of ths.
The diary technique presents so many difficulties that it will only be used
when our requirements can be met in no other way. What are some of these
requirements? One of them might be the timing of activities within the day,
when an hourly record is needed. Another is the summing of certain activities
The diary technique refers to a daily record kept by the respondent at the over a period of a week or longer for the same respondent, such as time spent
research worker's request - not to a document containing intimate personal in outdoor leisure activities, weekdaylweekend contrasts or the reciprocity of
confidences. The intention is to obtain, as accurately as possible, a daily or even visiting behaviour. A third requirement might concern such unobservable
hourly record of certain of the respondent's activities. Since the technique is behaviour as dream records. Sometimes better data might be obtained by
expensive, difficult to design and to place, and hard to analyse, it should only be specially stationed observers or by mechanical means - for instance, the daily
used when the necessary estimates cannot be obtained in any other way. tasks of teachers, or television channel-switching habits - but observational or
Moreover, there is the danger that the respondents' interest in writing up the mechanical methods present their own difficulties. Diaries come into their own
diary will cause them to modify the very behaviour we wish them to record. If, particularly when we are dealing with information that is not likely to be
for instance, they are completing a week's diary of their television-viewing remembered accurately for a period of time and needs to be recorded
behaviour, this may cause them to engage in 'duty viewing' in order to 'have immediately.
something to record', or they may view 'better' types of programmes in order to The design of a diary form will follow the same lines as any other self-
create a more favourable impression. completion device. We will have to pay a good deal of attention to definitions (for
Most commonly, diaries deal with behaviour rather than with attitudes, instance, in a health diary, what details pertaining to disease, malaise, self-
interests or emotions. Mass-media consumption patterns are a frequent topic, medication or medical consultation are to be recorded) and to the problem of
along with many other spare-time activities, illnesses, visiting and being visited, time intervals (daily, hourly, minute-by-minute). In setting out the form, more
the use of various forms of transport and so on. Diaries cover a given time span, attention than usual must be given to making it clear, providing adequate space
such as a week or a fortnight, or sometimes a much longer period. The limited and making it look interesting and important. As always, careful pilot work
time span is then regarded as a sample from which weekly, monthly or annual using alternative diary layouts is vital; it is, for instance, not at all easy to find out
rates can be calculated. This raises the question of the representativeness of such if the diary form is failing to obtain certain information. A frequent problem is
a time sample; how 'typical' was the respondent's behaviour in that particular the difference between weekdays and weekends in the patterning of many
week? Will the typical and the atypical 'average out' when we collect large activities.
numbers of diary records? Is the activity with which we are concerned subject A diary will need to be accompanied by written instructions. These should
to fluctuations? (Compare, for instance, newspaper reading, which shows generally cover the need for accuracy and frankness, the danger of allowing the
fluctuations between weekdays and the weekends but is fairly stable over the diary to influence the respondent's behaviour, and the importance of filling in
year, with gardening, which will show wide seasonal variations as well.) Much the diary at the requested intervals and not at the end of the period. Among
depends on the degree of generalization required; the figures may be typical for other topics, we may also deal with the respondent's feeling that a given diary
a given week or month but not for the year as a whole. For these reasons the period may be atypical.
dates or spread of dates for diary placement should be chosen with great care. Next, there are the problems of placement and collection, and the sampling
As we have seen (see Chapter 8), there are various types of questions from problems allied to these. Diary samples are often small, and the difficulties in
which we may derive estimates of respondents' behaviour. Here are some placement increase the danger of volunteer bias. Many investigators stress the
examples (I)'Within the past seven days, how often have you . . .'; (2) 'Within need for a personal touch, suggesting that an interviewer should spend time
the past seven days, have you done any of the following: (Tick all that are right persuading the respondents, instructing them in the details of the recording
for you) (a) (b) (c) (d) (e) (f)' (3)'HOWlong ago is it since you last . . . ?';(4) 'How process, perhaps calling in or telephoning once or twice during the diary period,
many times a week do you usually . . . 7' The type of question used will depend and personally collecting the diary and thanking the respondents. If co-operation
on the purpose of the research. For instance, the second question enables one to can be obtained, school children can be asked to fill out diaries about their
conceal the purpose of the study, while the last one asks the respondent, in previous day's activities for a week, under the (non-directive) supervision of
effect, to ensure that the answer is'typical'. All these questions present problems their class teacher, during the first period of every school day. They can even be
254 Quesfionnaire Design, Inferviewing and Aftifude Measurement Variefies and applications 255
asked to keep notes of their activities over the weekend for entry in Monday many now classic investigations. One of his earliest researches dealt with a
morning's diary. Some schools regard this as very useful practice for their pupils. holiday camp community of adolescent girls. The girls lived in fourteen cottages,
It is important, in such cases, to brief the teacher personally, to maintain the and there was a good deal of squabbling and unco-operative behaviour both
children's interest by means of letters or visits and to make it very obvious to the among girls sharing the same cottage and among girls in different cottages.
children that their diary records are being collected by the research worker and Moreno asked each of the girls to state - on a short, confidential questionnaire
will not be seen by the school. - the names of the girls in the camp with whom she would most like to share
In some studies it is possible to have one informant in a family who keeps a a cottage and the names of those with whom she would least like to share. By
diary for the whole household; for instance, in respect of leisure activities in the sorting the names into groups of individuals who were mutually attracted and
home or in respect of illnesses and medical needs. Care must be taken, however, gradually rehousing the girls accordingly, the disharmony in the camp was
that the chosen informant really is in a position to give full information on others geatly reduced.
and is not absent from the home for crucial periods or unaware of the activities This early application of the technique contains its most important ingre-
of some persons in the family. Other members of the family should also be dients, though a good deal has been added subsequently. Let us examine it in
warned against letting the diary-keeping affect their behaviour. some detail. First of all, the technique can only be applied to a group, although
Since diaries often continue for periods of a week or longer, we have the we may ultimately arrive at scores or descriptions for each individual in a group.
problem of sustaining motivation. It often happens that, as the diary period The group should be closed, in the sense that each member of the group can both
wears on, less and less information is recorded. Ways of preventing or reducing make and receive nominations, that each member knows at least some others in
this 'falling off' effect include encouraging letters, personal visits or telephone the group and that nominations outside the group are not permitted. The
calls, where possible. Others include varying the colours of the daily pages or by nominations express positive feelings of some kind (liking, admiration, utility,
adding extra questions now and then on separate slips of paper. The important wish-fulfilment), and the technique is often elaborated by asking also for
thing is that whatever the method of encouragement, we make the respondents nominations expressing negative feelings (with whom would you least like to
feel that we realize how much work we are asking for, that we are very grateful be?).The number of nominations is usually limited (three is a common number)
for their assistance and that the data really are important for the purpose of our and is the same for every respondent. The questionnaire should specify a
research. Perhaps, in the final analysis, how the respondents feel about the situation or criterion that the respondents should bear in mind when making
stated purpose of our study will be more important than personal encourage- their nominations: A criterion could be, as we have seen, sharing the same
ment or financial inducements. dwelhg, or doing homework together, or belonging to the same club and so on.
Diaries are among the most arduous techniques to code and process. They It is often suggested that better validity is obtained if the criterion is real, so that
provide rich illustrative material but, in addition, we usually need to abstract the respondents know that some social action will follow the analysis of the
certain data for tabulation. Let us take a very simple example: bedtimes. If we choice patterns, such as the reassignment of boarders to different dormitories.
have a week's diary for each respondent, then we have seven bedtimes available. It is also important that the responses be given privately.
We may code and process each of these separately to show mean differences of In its original form the technique is extremely simple. Allthat is required from
bedtimes between different days of the week, and between weekdays and each respondent is a slip of paper giving his or her name and the names of the
weekends. However, we shall also want to develop codes covering seven days choices. The situation to which the choices refer should also appear on each slip.
together for each respondent. For instance, we may want to develop a If these data are available for each member of the group, and no choices have
'regularity' code to show how much the bedtimes fluctuate in the course of a gone to persons outside the group by mistake, then the investigator is in a
week. These are simple tally codes; things become more complex if we have to position to draw a visual representation of the pattern of likes and dislikes within
assess, say, the 'sociability' of a family on the basis of a week's diaries, or their that group. Probably the easiest way to start doing this is to draw as many small
tendency toward eating health food, or the 'richness' of a person's dreams. We circles as there are members in the group and arrange these in the form of an
must bear in mind also that the diary is often part of a larger survey and that the oval. Taking each completed questionnaire in turn, we may draw arrows from
diary results must be linked to other data. Given the difficulties of design, the chooser to the chosen - using only first preferences, for a start (see Figure
placement, completion and processing, it is not surprising that diaries are often 13.1).The first attempt will not be very satisfactory, since it will show too many
given to only part of a sample. lines criss-crossing each other. The next step is to break the oval pattern and to
rearrange the results in a more meaningful way. For instance, we might start
with the most popular individual - the one receiving the largest number of
Sociometry choices. We may put him or her in the centre and arrange around this person
little circles of all the respondents who have chosen him or her and also the
The sociometric technique enables us to obtain a visual picture of the pattern of person he or she has chosen. Next, we take another very popular individual and
likes and dislikes among people in a group. The name J.L. Moreno has long been plot their choosers. After that, we draw around the periphery those who have
associated with the development of this technique and with its application in chosen one of the individuals surrounding these very popular persons and so on.
256 Questionnaire Design, Intemiewing and Attitude Measurement
I Varieties and applications 257
Matrix analysis starts by setting up a square of rows and columns with the
names or numbers of the group members both across the top and down the side.
customarily, the latter represent the choosers. In the row opposite each name
we may now plot this person's choices and rejections in the relevant columns.
First, second, and third choice may be indicated by entering the numbers 3,2 and
1 in the appropriate cells (or some other weighting or notation may be
employed); rejections may be similarly entered, preceded by a minus sign, or a
separate rejection matrix may be composed. Some of the data for index
construction may be readily compiled by adding up the entries in each column
representing the incoming choice patterns. For some purposes, we now proceed
to rearrange the matrix, by changing the order of the individuals simultaneously
and in the same way both in the rows and columns, aiming to bring as many
Example 13.15.
CHOSEN
Middle Class Working Class
A B C D E F G H I J K L M N O P
Figure 13.1. I 1
positive choices as possible near the diagonal, while moving rejections away strengthen the evidence for the greater group cohesion of the working-class
from the diagonal (see Example 13.15). This will reveal cliques, leadership boys.
patterns and group cleavages; but the clerical work involved is tedious unless a In many investigations we have other information available to us about each
computer program is available. individual, besides the sociometric choice data. For instance, we may have social
The x's represent the first choices of a group of schoolboys, divided according class, sex, age, ethnic group and so on. This will enable us to group the
to their social background. Although we have not yet plotted their second and respondents in a meaningful fashion and to study the extent to which such a
third choices nor their rejections, we can already see the beginnings of certain grouping accounts for the choicelrejection patterns, with the aid of matrix
trends. For instance, eleven out of the sixteen choices are 'within class', and only analysis. We can distinguish various kinds of in-groups and out-groups, and we
five are 'across class'; this suggests that social background might be important to can calculate the extent to which choices are restricted to the in-group and
these boys in deciding with whom to make friends. We also note that boy D is rejections to the out-group, and how many choices take place across subgroup
outstandingly popular; interestingly enough, all the 'across-classJ choices of the barriers. For instance, very young children in a mixed group will choose each
working-class boys go to him (although his own best friend, H, is a middle-class other without regard to sex; later on, boys and girls will only choose their own
boy). Could this mean that working-class boys are basically more inclined to stick sex, but toward adolescence there will be some choices across the sex line.
to their own class, unless an outstandingly attractive middle-class person, such Similarly, the degree of integration of minority-group members can be studied
as boy D, happened to be in the same group? We note further, that there are and compared.
eight boys who have received no first choices. Are these boys, in fact, isolated or The sociometric technique can also be used to focus attention on certain
rejected? To answer such questions we shall need to plot the second and third individuals. Thus, we may wish to identify the qualities that make for popularity
choices and the patterns of rejection, and the study will have to be repeated in or leadership in certain groups, or the effects of isolation or rejection; sociometry
a number of groups before more general conclusions can be drawn. will enable us to pick such individuals for more intensive questioning. In
In Figure 13.2, we see the same pattern of first choices drawn as a diagram. industry or in the armed forces, 'buddy ratings' have been used to study the
The 'class-barrier' runs across the page. As we have seen, boy D is very popular, relationship between leadership and popularity or to identlfy people who are
and there are five choices across the 'class-barrier'. We now also note another powerful and influential.
interesting phenomenon, namely, that among the working-class boys there are It is possible to construct a sociogram on the basis, not of questionnaire
two 'pairs' (reciprocated choices). This might, if repeated in further analyses, responses, but of observations of frequency and duration of contact between
members of a group. It is also possible to use the sociometric approach
projectively: the respondent has to answer such questions as: 'who do you think
would choose to work with Mr A?' and 'whom do you think Mr A would
choose?' thus giving us a consensual percept of relations existing between
members of the group.
Sometimes, no limit is placed on the number of names permitted; this enables
us to calculate a measure of social expansiveness of each individual. We can also
vary the criterion, for instance by asking first, 'who are your best friends?' then,
'with whom would you rather be, if you had to join the army?' and next 'whom
would you like to take home to tea?' This may tell us something about the
qualities that our respondent considers necessary in a friend, in a fellow soldier
and in a visitor to take home to the family.
middle This last elaboration raises the question of validity. Some investigators have
class suggested that sociometric questionnaires are of doubtful validity unless the
respondent knows that the results will be used for a restructuring of the group.
working Many others, however, have freely used hypothetical criteria in ways analogous
class to an attitude questionnaire.
Network analysis (see Knoke and Kuklinski, 1982) is a technique which, in
many ways, is analogous to sociometry. It is used to study the relations and
communication patterns within and between members of various groups.
Epidemiologists, for example, find it useful for the modelling of disease-
transmission patterns.
Figure 13.2.
260 Questionnaire Design, Interviewing and Attitude Measurement
Selected readings
RATINGS
Guilford, J.P., 1954, Psychometric Methods, McGraw-Hill, New York.
DATAPROCESSING
Arguably the best discussion on rating scales is still in Chapter 1.1of this book. For
paired comparisons see chapter 7.
Postlethwaite, T.N., 1962, Young Europeans in England, Political and Economic Planning
Foundation, London.
SEMANTIC
DIFFERENTIAL AND REPERTORY GRID
Osgood, C.E., Suci, C.J. and Tannenbaum, P.H., 1957, The Measurement of Meaning,
University of Illinois Press, Urbana, Ill.
The main source book for the semantic differential technique and its various At the beginning of the study we should explore what data-processing facilities
applications. will be available to us and what computer programs and equipment we shall be
Fransella, Fay and Bannister, Donald, 1977, A Manual for Repertory Grid Technique, using. Certainly it would be most unwise to leave such considerations till after
Academic Press, London. the fieldwork, for by then it will almost always be too late. It is best, therefore,
Has become a standard text on the repertory grid. to seek advice on data processing, and on computers and computer software, at
a very early stage so that the data-processing requirements can be incorporated
Kelly, George A., 1955, The Psychology of Personal Consfrucfs, W.W. Norton, New York. from the outset, thus enabling the analysis to proceed as smoothly as possible.
Volume I contains the basic theory and the development of the repertory grid test. It is wise to anticipate the data-processing requirements also during the pilot
Bannister, Donald and Mair, J., 1970, The Evaluation of Personal Constructs, Academic Press, stage. If it is intended, for example, to place data-processing indications and
London. numbers in the right-hand margin of each questionnaire page, perhaps in a
Entertaining early text. Useful on reliability and validity of repertory grids. column headed'for office use only', then this layout should be introduced during
the pilot work to check that it does not unduly distract the respondents. As we
SOCIOMETRY have seen (see Chapter 4), such data-processing indications should be placed
close to, and aligned with, the answer categories or boxes to minimize transfer
Moreno, J.L., rev. ed. 1953, Who Shall Survive?,Beacon House, Washington DC. errors. For some questionnaires it is possible to generate bar codes for each
This has long been the chief source book of the sociometric movement and a response category, or to make the responses machine-readable in some other
recognized classic. way, so that the data can be directly entered into a computer.
Moreno, J.L., 1960, The Sociomefry Reader, The Free Press, Glencoe, Ill. During the quantification stage of our survey we have to face up to the fact
A source book of sociometric research. that, however subtle our questions and however thoughtful and cooperative
Knoke, D., and Kuklinski, J.H.,1982, Network Analysis, Sage. our respondents have been, the main purpose of the questionnaire, and of the
survey as a whole, is measurement. The final product is likely to consist of a series
of tabulations and statistical analyses; these, together with a few illustrative
quotations from the raw data, will be turned into a report showing in what ways
our findings bear on the hypotheses with which we began. During the
quantification stage the words that were spoken or written by our respondents
will be processed; they will be turned into figures and symbols that can be
counted and added up. In this way we obtain the entries for the tables that we
need in order to draw conclusions.
How is this to be done? How are these bundles of questionnaires or interview
records to be transformed into meaningful figures which can be used for
tabulation?
Essentially the processing routine will be determined by the nature of the data
collection techniques, though something will depend on the size of the survey,
the composition of the questionnaire, the processing facilities available and the
262 Questionnaire Design, Interviewing and Attitude Measurement Data processing 263
the data-processing stage. The procedure is straightforward enough, but it helps MARITALS or whatever we like. To avoid confusion, we should try to generate
to observe some systematic data-processing conventions. Let us take, as our first labels which are easily understandable by fellow researchers.
example, a very commonly asked factual question:
by using a double-blind procedure or by lodging the only existing list of names wasteful because they require re-coding of questionnaires that have been coded
linked to case numbers with a colleague abroad. already.
Use can be made of the identifying case numbers for classification purposes, How do we set about designing a coding frame? Probably the first step should
by linking them to the sampling frame in some suitable way. Suppose, for be the examination of a representative sample of responses. In practice, we select
instance, that we have a sample of children selected through their schools. We a convenient number of questionnaires (say, fifty or sixty cases) on a
may decide to give each child a six-figure case number: the first digit might representative basis and copy all the responses to a particular question on to
indicate sex (I,or 2, meaning girl or boy); the second digit might indicate district sheets of paper, or on to a word processor. At the top of each page will be the text
or area (assuming there are not more than ten districts represented in our of the question as it appeared in the questionnaire, and below that will be copied
sample); the third digit will indicate the school that the child attends (assuming all the various answers given to that question by our subsample of fifty or sixty
that we sample not more than ten schools in each district);the fourth digit might cases, each answer preceded by the case number. Where answers are few, for
be used to indicate the form or level within the particular school; and the last two instance if the question applied to only part of the sample, more cases will be
digits will identify any given chld within his or her class. Thus, case number needed, until we have a sufficiently large and varied selection of answers. When
279513 would mean, to the initiated, a fifth-form boy in school 9, in district 7, we come to design the coding frame of the next free-response question, we go
identifiable as number 13 on the list of names in his class. Such a system would through the entire batch of selected questionnaires again, copying all the
also enable us to bring together all the third-formers in our sample, or all the responses to that particular question together, so that we can look at them.
children from area 4, or all the girls from school 81 (meaning school 1in district From this point on we must bear in mind very clearly what it is that we are
8) and so forth. There is no need to insist on numbering all cases consecutively; trying to do. By imposing a set of classificatory categories, perhaps eight or ten
if there are, say, only three school districts represented in the sample, then the in number, on a very much larger and probably very varied set of responses, we
remaining numbers in that variable will be left unused. are inevitably going to lose information (see Chapter 7). Bearing in mind the aims
It is often wise not to leave the design of a case-numbering system until the and hypotheses of the survey and the particular purpose of the question under
data-processing stage. For example, postal surveys often put case numbers on consideration, we must so design the coding frame that this loss of information
outgoing questionnaires (unless we have promised our respondents complete will occur where it matters least, enabling us to run our comparisons or test our
anonymity); interviewers are also sometimes given sets of case numbers to hypotheses with the greatest accuracy. This means that our set of categories will
assign to respondents. The design of a case-numbering scheme will force us to not necessarily be designed simply 'to do justice to the responsesJ; other
think about the intended statistical analysis, and will help to keep track of the considerations may apply, and compromises often have to be made.
questionnaires during the data-processing phase. The case numbering system For a start, how many categories should we have? If there were no
we adopt should appear in the code book. constraints, and we were anxious not to cause any distortion, we might like to
have almost as many categories as there are responses, grouping under one
Coding frames heading only those responses that are identical. This is obviously not a practical
proposition. Even if we could afford to design so elaborate a coding scheme, we
Each free-response question, probe, sentence-completion item, or other 'open' would,probably find during the statistical analysis that each category contained
technique in our questionnaire will require its own classification scheme. Only only one case or a very few cases. Therefore, the number of categories we can afford to
rarely can we use a scheme devised for some other inquiry. Our first task will have will in part be determined by the number of cases in the sample and the number of statistical
therefore be the design of all the classification schemes, usually known as 'codes' breakdowns we shall use; a category that will, in the final analysis and after
or 'coding frames', required for our particular study. Quite possibly we have subdivision of the sample, hold fewer than two or three dozen cases must
already given this matter some thought during the pilot stages, and perhaps we usually be regarded as a luxury. However much it offends our sensibilities, we
have produced some tentative drafts for coding frames. However, the coding must realize that it is pointless to retain a category that is used by too few people.
frames proposed during the pilot stage will have been based on the raw data There is an exception to this argument. It sometimes happens that we have a
obtained from pilot samples, and since the responses of such samples may differ hypothesis about a certain type of response being absent or very rare. In that
markedly from those of the main sample, we must always construct our coding case we might reserve a category for it, in order to show just how rare it is. For
frame with the aid of responses from the main sample. instance, suppose we had asked people why they had changed from one brand
It is very tempting to start the design of our coding frames as soon as the first of cigarettes to another, and suppose, further, that we wished to test the specific
returns from the fieldwork arrive back at the office. However, since these are hypothesis that people very rarely admit to reasons of economy in answering
likely to be unrepresentative, it is advisable to wait until all the returns, or a such a question. In such a case we would make up a coding frame suitable to the
substantial part of them, are in. Otherwise, we run the risk that the coding distribution of the answers we get but, come what may, we should reserve one
frames will turn out to be a poor fit, requiring amendment, perhaps several category for 'economic reasons' or something like that.
times, in the course of the coding operation. It is always undesirable to have to Most survey practitioners try to keep their coding frames down to twelve to
amend a coding frame in midstream, since amendments produce errors and are fifteen categories, though they may go beyond this number occasionally. Since
Data processing 269
268 Questionnaire Design, Interviewing and Attitude Measurement
coding frame requires a logical structure; the classification of favourite subjects
we will require categories for 'miscellaneous', 'don't know' and 'no answer', this at school might be an example. Here we would perhaps wish to use two linked
would leave us effectively with nine to twelve categories. This number may, at frames. The first frame would have broad categories such as (1)'languages', (2)
first sight, seem impossibly small when the variety of responses is being 'numerical subjects', (3) 'natural historyJ and so forth, while the second frame
considered, yet it is surprising how often one starts off with a much more would be different for each of the broader categories and would contain the
ambitious scheme only to find that up to a dozen categories will suffice after all, subcategories, so that, say, code 14 might be French, code 26 might be
with the added advantage of increased frequencies in each category. trigonometry and code 36 could stand for geology.
What other considerations guide us in the composition of coding frames? Let For some questions, typically those used for classificatory purposes, there are
us take, for example, a question asking for the respondent's favourite film star. probably some well-designed and elaborate coding frames available ready-made.
Let us assume that we have copied the replies of five dozen people and that we A classification of occupational prestige might be one example. When using
are now faced with the ~roblemof classifying these responses. One approach prepared coding frames one should follow the coding instructions most
might simply be by frequency. We allot, say, seven categories to the seven names carefully, and, of course, the original question(s) should be the same, if
occurring most often and lump the remaining names in one or two other comparability is desired with the results of other investigations using the same
categories. Or perhaps we wish to expand the frame; we could have dozens of coding frame. Occasionally, too, we may have asked the same question in 'openJ
categories, each with one film star's name, if we chose. Or we might decide to and in 'closed' form; in that case there may be something to be said for using at
group the names under different labels, such as 'romantic', 'Western', 'musical' least some of the pre-codes of the 'closed' question as categories in the coding
and so on, according to the type of role with which such film stars are associated. frame of the 'open' question.
Then again, we may decide to have two coding frames, one for male and one for Every coding frame is likely to need two or three categories that are standard,
female film stars. Or we may wish to group together those stars who also appear namely 'miscellaneousJ,'don't know' and 'no answer' or 'not ascertained', though
on other mass media, and those who do not. We may classify stars by their ages, the latter two categories are frequently grouped together. On the other hand,
or by their ethnic background, or by the number of divorces they have had. So sometimes it is important to know how many respondents said that they did not
we see that it is often not a simple matter to design a coding frame that will 'do know the answer, or which ones refused to commit themselves; these two
justice to the data' and, moreover, the type of coding frame we need will depend categories may not just be 'waste categories'. Into 'miscellaneous' go all those
on what we wish to find out. responses that cannot readily be fitted into one of our prepared categories. In
Suppose, for instance, that we wish to examine the hypothesis that men will cases of doubt, it is better practice to classify a response as 'miscellaneous' than
most often admire female film stars, whereas women will more often mention to force it into another category. One reason for this is that it is best not to blur
male film stars. In that case, all we need to do, strictly speaking, is to classify the the limits of the categories.Another is that if such doubtful responses occur with
responses by sex into just two categories, male stars and female stars. This unexpected frequency, then at some point they can be 'rescued', by making the
would tell us all we needed to know - though it would not enable us to go very decision to amend the coding frame by introducing a new category. In that case
much further. On the other hand, suppose we had the hypothesis that a lot we merely have to look again at the responses coded 'miscellaneous' with a view
depends on the star's age in relation to the respondent's age, with younger to reclassification, instead of having to recode every response category. Such a
respondents admiring a somewhat older and more mature actor or actress, course of action should be seriously considered if the frequency of 'miscellane-
while middle-aged respondents prefer a younger star. In that case we would ous' responses rises above, say, 15 per cent or so.
need a fairly complex coding frame giving categories of age differentials, up or It should be realized that code categories can always be combined, for example
down from the respondent's own age, and to do the coding we would need to by putting together all the male film stars, or all the favourable plus moderately
know both the respondent's age and that of his or her most admired film star. favourable responses, or all the respondents doing manual labour of any kind.
When we copy out the responses, it is helpful to group the respondents in This is sometimes necessary when we are dealing with small sub-analyses,
terms of one or more variables of concern, such as sex, age, social mobility and where the lack of cases is making itself felt.
so on. This often suggests differences in content, flavour or expression between Each category in a coding frame should be designated in the clearest possible
subgroups, and a coding frame can be designed to highlight these. For this way. It should be described in words, or given a label, and it is always helpful to
reason, the copied responses must not merely be regarded as a try-out; they give many illustrative examples taken from actual responses. Suppose we have
should be most carefully studied and perused. asked people a question about the platform of a given political party, and that we
Usually, the order of the categories is unimportant, and the categories are wish to classify the answers in terms of the amount of knowledge revealed by
quite independent of each another (they form a nominal variable). Sometimes, the respondents. In such a case it would not be enough to set up a coding frame
however, we may need a coding frame that is more like a rating scale. For with categories such as 'very good', 'adequate', 'poorJ and so forth. Obviously,
instance, we may use some of our categories to indicate the degree of this would lead to inconsistenciesamong the coders and might not be clear to our
favourableness with which the respondent views a certain person or object. readers. We have to set up definite criteria such as: 'very good: gives at least
Some responses would be classified as 'highly favourable', others as 'moderately three different items of party policy correctly', together with some examples of
favourable' or as 'favourable with reservations', and so on. Sometimes the
270 Questionnaire Design, Interviewing and AffifudeMeasurement Data processing 271
actual responses. This is particularly important when numerous coders will be of sergeant was or was not to be included in that category. Clearly, for other
engaged on the same survey, in order to ensure consistency and reliability. Even purposes one would code the material differently, in terms of home service
where the investigator does all his or her own coding, the categories should be versus various overseas theatres of service, or in terms of army, paratroopers,
as clear and as unambiguous as possible, for it is only too easy to change air force or marines, and so on. For some purposes we can make use of a special
standards as one goes on. It is also necessary that the reader know what is the marker, say, for all those with any kind of overseas service. Such a marker code
precise meaning of each category; often categories are ambiguous, but examples would enable us quickly to locate those respondents who had served abroad.
can make the meaning clear. Note, finally, that this is also an example of a logical code, which could largely
In the entire coding operation it is important to introduce frequent checks, for have been anticipated except for the cut-off points for each category.
statistics based on inconsistent coding can be very misleading Some coding Next, we look at a somewhat similar example. Here is a coding frame, from a
frames are relatively objective and merely require consistency and attention to student survey, for the answers to the question: 'During the past seven days,
detail on the part of the coder for instance, the coding of favourite school how many hours have you spent reading newspapers?':
subjects. Other coding frames, however, require a certain amount of interpre-
tation on the part of the coder; for instance, coding the job dissatisfactions of none
teachers, or the reasons people give for not travelling abroad more than they do. less than 1hour
We then have to face the delicate problem of designing a coding frame that goes 1hour to 2 hours 59 minutes
'deep' enough, yet one that can be used consistently by the coding staff available, 3 hours to 4 hours 59 minutes
bearing in mind their training and experience. In some investigations it is 5 hours to 6 hours 59 minutes
necessary to check every coded response, or to have two coders working 7 hours to 8 hours 59 minutes
independently on the same data and then discussing and resolving the 9 hours to 10 hours 59 minutes
differences between them. The primary aim must be consistency and the 11hours and over
elimination of ambiguities; a coder who 'stretches' a category, in order not to 99 no answer, not ascertained
have to classify a response under 'miscellaneous', or one who 'improves' a o don't know, can't remember etc.
category from one day to the next, or who 'knows' what the respondent 'really
meant', merely does the study a disservice. The better and clearer the coding Here we have another ordinal code, going up in regular steps of two hours.
frame, the fewer such occasions will arise. Obviously, the actual categories have been determined from a sample of
responses. How else would we know that intervals of two hours would be the
We may now turn to some examples of coding frames. Here, first, is a very most appropriate, or that very few students read newspapers for more than
simple code that enables us to classify a man's army service rank: eleven hours a week? A 'miscellaneous' category is not, of course, required, but
note that 'none'has been given a separate category on its own - to show which
no military service students had not read any newspapers at all.
private We also note the special cari that has been taken to create non-overlapping
noncommissioned ranks, below sergeant cafegories(as was the case with the preceding example).All code categories musf be non-
noncommissioned ranks, sergeant and above overlapping. Beginners frequently make the mistake of creating categories that
commissioned ranks up to and including captain overlap, in part. This becomes most obvious in numerical codes such as the one
commissioned ranks, major and above above, or in the following age categories: 20-30,30-40,40-50 etc. (The problem
special service troops, marines is: how should a respondent aged 30, or 40, be coded?) In non-numerical codes,
navy or air force, merchant navy the problem of overlap is less obvious, but the danger is always there, and can
no answer, not ascertained, rank not specified lead to inconsistency among coders.
don't know, can't remember etc. In the same survey of student reading habits, some questions were asked
about the most recent textbook that the student had read in the college library.
Obviously this is a fairly crude code, designed for a particular purpose - chiefly, One of the questions was 'What system did you follow in reading this book?' to
that of ascertaining the highest army rank attained by the respondent. Most of which the following coding frame was applied:
the categories are, therefore, prestige levels, and the categories are ordered,
though this is not strictly necessary here. Note that, for special reasons, a 1 miscellaneous
separate category has been reserved for private; also, that most of the other 2 chapter headings
categories cover a range (from - to) of ranks, so that the highest and lowest 3 skim chapter by chapter, then make notes
rank have to be specified. Thus, if we had worded category 3 as 'noncommissi- 4 read from cover to cover
oned ranks up to sergeantf, this would not have made it clear whether the rank 5 skim and make notes
272 Questionnaire Design, interviewing and Attitude Measurement Data processing 273
6 concentrate on one section only 12 classical, other
7 use index, look up pages o no novel mentioned
8 read chapter summaries
10 just read conclusions I
1
(NB: Classical meant that the book was published before 1914.)
II introduction only
12 read various sections thoroughly This is an example of a double code, and an example of a grouped code as well.
99 don't know, no answer, not ascertained To begin with, a relatively crude classification was made of the main types of
books mentioned by students - a classification that cannot bear comparison to
We observe that the categories for 'don't know' and for 'no answer; not the elaborate systems of classification employed by librarians, but that sufficed
ascertained' have been grouped together. We also notice that the categories are for the purpose at hand. It then became evident from the study of a sample of
described in somewhat abbreviated style; this is liable to lead to misunderstand- responses that novels were by far the most popular type of extra-curricular
ings. Likewise, it is not clear how categories 3 and 5 differ from each other; this reading. It was therefore decided to do a more detailed, but separate,
kind of ambiguity should be avoided, and it should certainly be one of the points classification of this kind of fiction. Any respondent who mentioned a novel was
to be discussed with the coders before they start. The code is obviously based on given 11on the first coding frame, and coded once more on the second frame.
the study of a sample of responses; it is neither ordinal nor logical, in our sense, In the second frame we have, first, three broad groupings (modern, between
To some extent, the makers of the frame were also guided by the knowledge of wars, and classical), and then, within each of these, four categories (romantic,
what the teaching staff regarded as good practice in getting the most out of time historical, novel of ideas, and other). The relative crudeness of both these frames
in the library. Last but not least important, this is a multiple-mention code: a illustrates once again the inevitable compromise between the degree of
student can obviously use two or more of these systems in reading a book, and refinement in which one can indulge and the constraints of sample size, number
we must so devise our code that we can cope with multiple answers (see below). of statistical breakdowns, quality of coders and so on. Both frames also illustrate
Also in the same study of student reading habits were questions about the the need for examples: in the actual coding operation lists were kept of names of
most recent extra-curricular book read. These books were coded as follows: books that occurred frequently, together with their date of publication and
agreed classification by content. Even so, there would obviously be much room
miscellaneous for discussion and need for checking and cross-checking. In the end, if it seemed
biography, autobiography as if a particular book could fit into more than one category, it would be
travel and geography necessary to lay down a ruling, however arbitrary, in order to maintain
crime, detection, Westerns, mysteries consistency among the coders.
poetry We now come to a more difficult kind of code. In an attitude study of teachers,
essays and short stories the following question was put: 'In these days, what qualities in children do you
humour think a teacher should encourage most?' This question is obviously very wide
indeed, and the answers covered a considerable range. Here is the frame that
plays was finally developed:
books about art, music, ballet, theatre etc.
theology, religious books
novels (see next column also) religion
no answer, not ascertained, don't know ambition, striving
self-development and happiness: spontaneity, happiness, curiosity, creativity,
NOVELS CODE
self-reliance, active and open personality, originality
rational use of energies for the attainment of educational and work goals: industry,
modern, romantic efficiency, perseverance
modern, historical active adjustment to social situations: sociability, co-operativeness, comradeship
modern, novel of ideas (social message) inhibitory adjustment to social situations: self-control, self-discipline, correct
modern, other manners, cleanliness, orderliness
between wars, romantic inhibitory adjustment to authority figures: respect for authority, deference to
between wars, historical teachers and elders, obedience
between wars, novel of ideas (social message) self-assertive adjustment to social situation: competitiveness, toughness
between wars, other goodness, kindness, love, tolerance, generosity
classical, romantic adherence to enduring values, personal integrity: truthfulness, honesty, sense of
classical, historical justice, sincerity, sense of honour, dignity, courage
classical, novel of ideas (social message) other
no answer, not ascertained, don't know
274 Questionnaire Design, Ideroiewing and Attitude Measurement Data processing 275
This coding frame requires a good deal of interpretation of the responses before
they can be classified. The coders would have to think and discuss quite carefullully; i dealing, perhaps, with several surveys at a time, and obviously some control
arrangements have to be made in order to regulate the various responsibilities
inevitably, coding this kind of material is slow. Note that often a broad verbal
label has been given, followed by some examples. We observe that this, too, must
1 and the flow of work. In particular, there should be a standard procedure for
keeping track of each questionnaire from the moment it comes into the coding
be a multiple-mention code (see below). I section, for it is very easy for questionnaires to get lost, if they happen to become
query cases, or part of a try-out sample, or if they get mixed up with a batch of
Finally, here is an example of a frame-of-reference code. The question asked was:
'Taking all things into account, what would be the most likely cause for another I questionnaires from another survey.
world war? I don't mean the things that happen just before a war, but the real
cause.' Here is the coding frame: 1 Usually, it is best to let- each coder work right through each questionnaire,
going from question to question and from code to code until the final page, to
may have missed out one or two whole pages, or stopped part of the way
through the questionnaire. But in addition there may be many non-respondenis; i statistical significance.Multi-variate analysis is likely to require the creation of a
correlation mairir; that is, a square table containing the correlation of each
that is, people who are missing from our data set altogether, perhaps because
they could not be interviewed or because they did not return a postal I variable under consideration with every other variable in that set (in other
words, two identical triangles, so that every correlation coefficient appears
questionnaire.
Every survey researcher will be confronted by the need to take some
1 twice). This may call for some complex pairwise deletions. Let us imagine, for
example, that we intend to do some factor-analytic scaling (see Chapter 11)on
uncomfortable decisions about missing data. Obviously the best way to I a couple of dozen attitude statements. This will require the calculation of 276
minimize such problems is by making every effort to prevent them - by
reducing the non-response rate to the lowest possible figure. However if, despite
our best efforts, many respondents have refused, or been unable, to answer a
I1 correlation coefficients (24 x 23 divided by 2), each of which may be subject to
sample 'shrinkage' (pairwise deletion), and a changing sample size N, to a
different degree!
certain question, should we discard it? If a person has done the first part of our Ideally, we should want to have an 'all-completed' data set for the entire
questionnaire but has left the rest uncompleted, what should we do? Are there
any steps we can take to retrieve missing respondents? In thinking about such
I sample but, if that is not possible, then we need to make sure that our statistical
software package can cope with varying case numbers. If the loss of cases is not
issues we should be less concerned with reduced sample size than with the ever- I substantial and the possibility of introducing a bias is small, then we could, with
present risk of bias (see page 106): are people who have drinking problems less,
or more likely to refuse to be interviewed about their alcohol consumption and I the aid of listwise deletion, create a sub-sample which contains only cases that
have completed every item in the questionnaire, but there is always the
related topics? Is our political postal questionnaire more likely to be discarded by
the followers of a particular party? We must remember (see page 106) that even 1 possibility of introducing an inadvertent bias.
distributions we require, and any further tables will be mainly in the nature of to plan our before-versus-after set of comparisons.
elaborations. But if we are dealing with a more complex, analytic research Many statistical analyses will go no further, but with some studies a multi-
design, then the real interest of the survey analyst will lie in the interrelation- ~ariatestage will need to follow -or may be needed at an earlier stage. In a multi-
ships between certain variables, and the over-all distributions will chiefly be used variate analysis we try to explore the patterns of relationships among our
to plan the study of these interrelationships more carefully. variables not just in two-way, cross-tabulated form (for example, income level
In the case of a simple random sample, sampling errors and confidence limits versus possession of a video recorder) but in more complex ways, using many
of estimates based on the whole sample (say, of the proportion of respondents variables. To follow the above example, and perhaps having found a relatively
whose preference is for party P) can now be calculated. If interest focuses on weak association between income level and owning or hiring a video recorder,
differences between sub-groups (say, on the proportions of men and women we may cast our minds wider and start to wonder about income level and the
who prefer party P), then estimates of the sampling errors of such observed possession of consumer durables generally (for which we would have to
differences need to be calculated (see page 43), since these are likely to be higher. calculate a composite index), about income level and materialistic life-style in
As we know (see page 159) such estimates are further influenced by other ways (for example a second home, holidays abroad) and even about income
measurement unreliability and by response bias. level and 'materialism' as an attitude or value construct (assuming we have, or
For the study of bivariates (including cross-tabulations, breakdowns and can create, a measure of the latter). We are now embarking upon a more
different measures of association between two variables) we must turn to our exploratory stage of data analysis. Statistical techniques do exist to make such a
design. We should remind ourselves which are our dependent and our multi-variate analysis possible (for example multiple regression, principal-
independent variables and set up some kind of groupings within the sample that components analysis), but it would be best to plan these at the research design
will enable us to determine the relationships between them. Perhaps we should stage; that is, at the beginning of our study, not at the end.
first of all like to satisfy ourselves that we have a robust, reliable and valid In educational research, to take another example, we may try to 'explain'why
dependent variable; for example, achievement in mathematics (adding up marks some children do better at a given school subject (say, mathematics) while others
for arithmetic, geometry, algebra and so forth and calculating each student's do less well. Our dependent variable would be a set of maths test results for each
average over several semesters), or a social adjustment index (similarly a child, but our independent variables would be many: perhaps a measure of
composite score with several sub-domains). In more complex investigations intelligence, a measure of homework supervision, the number of children in the
there will be more than one dependent variable; it can also happen that a class, availability of teaching and computing equipment, parental occupation (for
dependent variable in one context becomes an independent variable in another example, parents with maths-related jobs), maths and other qualifications of the
part of the analysis. teacher, the child's intended career and further education, possession of an
After that, we shall essentially be engaged in making comparisons between electronic calculator, and so on. All of these, and many other measures, could be
fairly stable sets of sub-groups in our sample (for example those with a high simultaneously related to achievement in mathematics, and we could ascertain both
social adjustment score versus those with a low one) in relation to each the relative contribution of each variable to the child's achievement and their
independent variable, in accordance with our theoretical expectations as various combined 'effects'. In particular, since many of these variables overlap
embodied in the research design. A carefully devised basic plan is therefore with each other, we could show which ones contribute relatively little to the
needed for the generation of statistical 'production runs'. Of course, the better 'explained variance'once others have been included in the pattern, and which ones
and tighter our research designis, the more readily the analysis will 'run itself'. still contribute independently. We could also combine these variables into groups,
If, for example, we have a factorial design or one using multiple regression, then or blocks (for example, family variables, classroom variables) to show the relative
the main classifyingvariables are known from the start. Or, if we have employed importance of each group of variables compared to that of other groups.
a control group, then a matching operation will have to take place and group Obviously such an analysis will require a great deal of thought and planning.
univariates compared, after which comparisons between experimental and Part of a multi-variate analysis may involve us in 'building up' rather than
control groups can proceed. Similar preparatory operations will be required to 'breaking down' the data. Carrying out a principal-components analysis or
set up a run of comparative tabulations in the case of a before-and-after type of developing a new scale or a composite index would come under this heading.
design. If we are doing, for example, a quality-of-life study on a group of patients Quite often such newly generated variables will then require us to undertake
before, and some months after an operation, then we shall first need to compare new 'production runs' to show how these new variables are related to responses
the second ('after') set of data with the first, to see how many cases have been to other parts of our questionnaire. Before doing so, it is essential to calculate the
'lostJ in the sense of 'cannot be traced.'. Next, we shall need to compare the 'lost' reliability of any scale or index (see page 159) to make sure that it is at least
cases with the remaining ones on some of the most important variables in the adequate; if the reliability coefficient falls below .SO then further work using this
first ('before') data set to ascertain bias in the remaining sample of responders. variable must be regarded as of dubious value. The generation of new variables
After that we shall need to create a definitive data set which will include only at this stage is likely to present us with serious problems of interpretation, but
those cases for which we have complete follow-up responses, as well as a at least we can hope to replicate these new measures in subsequent studies.
complete set of responses from before the operation. Only then can we proceed The fourth and final stage of statistical analysis is difficult to plan. This is the
284 Questionnaire Design, Interviewing and Attitude Measurement Statistical analysis 285
analysis of special sub-groups. It comes after the main results of the survey have Statistical comments
been tabulated and digested, and have begun to give rise to new research
questions. At this point we shall very likely wish to undertake a series of Competence in statistics is a necessary requirement for the research worker in
additional breakdowns, cross-tabulations or correlational studies that were not any field; readers who are uncertain of their skill in this respect may try the test
part of the main analysis. Often such cross-tabulations require the identification in Appendix I. There is no shortage of textbooks in this field, covering
of a particular sub-sample (for example, people who are self-employed and are probability theory, tests of significance, and correlational and multi-variate
working from home; people who never vote; people who used to buy a particular techniques. A variety of computer software packages both for PCs and for
product but have ceased to do so) for special study. Or we may wish to explore mainframes are available, the most helpful of which will also be interactive; that
a pattern of relationships within several graded sub-samples, for example is, tables can be generated, inspected, changed and printed on line. Often such
studying the relationship between family size and maternal strictness within packages will automatically apply several tests of significance and measures of
each social class in turn. This kind of analysis will require us to 'hold constant' association to each table (not always appropriately). The facilities for creating
other variables, or to engage in the compilation of matched sub-samples. and re-shaping two-way tables would be one of the criteria by which to judge the
This part of the analysis is both interesting and time-consuming, and in a suitability of a statistical package for our proposed analysis.
sense there is no end to it, for there always seem to be further interesting Statistical techniques are like excavation tools and building blocks: they help
possibilities just over the horizon. Most surveys are never 'analysed-out' us to 'dig' into the data and mine them for precious findings, and enable us to
completely; the analysis stops for lack of money or lack of time. summarize and compose these into a meaningful structure, but, depending on
As we have seen in Chapter 2, unless this kind of detailed sub-group analysis the nature of the data, different statistical tools will have to be used for different
is planned for from the beginning and the sample collected accordingly (for purposes.
example, by over-sampling the sub-groups of concern), we tend very soon to When planning the statistical analysis of a survey, an important distinction
run out of cases. Perhaps this is just as well, for there are distinct disadvantages will have to be made between metric or interval-type data, and nominal or
in going on a 'fishing expedition'. The chances are that we shall end up with a categorical data (see Chapter 9), since they require different statistical treat-
great many wasted tables, with a few substantial and interesting differences that ments. Internal-type data have additive properties and equal intervals along a
we will be unable to interpret, and with a large number of small trends that may continuum. In social surveys variables such as age, family size, income, years of
whet our curiosity but are unlikely to emerge again in subsequent surveys. We residence and number of cigarettes smoked per day are all examples of interval
might find, for example, in a study of airport noise, that the people most likely scales. The statistical techniques applicable to them are means, standard
to complain about aircraft noise are those with marital problems. This looks deviations, t-tests and F-tests, regression methods, analysis of variance, product
interesting but, unless we can say how such an association has come about and moment correlation coefficients, and so on. Nominal data are not measured along
thus derive a new explanatory variable that might have wider applications, such a continuum; they lack additive or even ordinal properties and can best be
results are of doubtful value. Up to a point such exploratory analyses are useful thought of as frequencies in discrete categories, placed in no particular order.
and may yield important supplementary findings, but they need to be 'theory- Nominal data in social surveys might be marital status, place (statelcountry) of
driven' and we must learn to curb the urge to do too many. We should also birth, eye colour, religious affiliation, daily paper read most frequently, favourite
remember that at, say, the 5 per cent level of statistical significance, one in every school subject, or the reasons for likingldisliking any specific product. Applicable
twenty cross-tabulations will be 'statistically significant' by pure chance. statistical .techniques are percentages, Chi-squared tests and other non-
It is often the case that the over-all plan of our analysis will also determine the parametric devices (see page 157). In addition, between the interval measures
shape of the final report, and so it is perhaps worth mentioning that only a small and the nominal classifications lie variables that have ordinal properties such as
fraction of all our tabulations will eventually be published, perhaps less than 5 ranking data, measures of relative interest or preference, orderings of priority or
per cent. This 'wastage' is inevitable; many results can be summed up in a few prestige, and others (155). A number of statistical techniques such as Spear-
words, quoting perhaps one or two figures for illustrative purposes, without man's Rho and Kendall's Tau have been specially developed to use with ordinal
publishing an entire set of tables. This is especially true if the findings are data.
negative. If we have a questionnaire with 114 questions and we wish to find out It should be stated that the paramefric techniques of statistical analysis
which of these are related to the respondents'religion, then this will require 114 applicable to interval scales have considerably greater power and versatility, and
tables, none of which will be published if the results are altogether negative; can make fuller use of the information contained in our data, than non-parametric
such findings are likely to be summarised in a few words, unless they are of techniques.
special importance. When writing a report we must constantly ask ourselves: 'Is A typical social survey is likely to contain a mixture of interval measures and
this table really necessary?: otherwise the reader will not be able to see the wood nominal data, and these will have to be treated in different ways. This
for the trees. fundamental distinction will greatly affect the production of the tables in our
statistical analysis. For example, suppose we want to know whether religious
affiliation is related to alcohol consumption. We cannot answer this question by
286 Questionnaire Design, Intemiewing and Attitude Measurement Statistical analysis 287
calculating a single measure of strength of association, such as a correlation scale? O r what about five-point Likert scale items in attitude scales (see Chapter
coefficient, because in the case of the usual product-moment correlation both II)? Is it justifiable to treat these as metric variables? And how can we justify
variables must be metric, whereas in our example one variable (alcohol making similar assumptions in the case of semantic differential items (see
consumption) is metric but the other (religion)is not. So how, by what means, Chapter 12), or inventories (see Chapter IS)? The temptation to do so is often
could we explore and present this relationship (if any), and what instructions strong because the statistical 'mining' tools for metric variables are more
shall we give to our computer? Having drawn attention to the general problem versatile and more powerful.
of trying to relate metric to non-metric data, we can say that some possible Sometimes, however, we prefer to carry out transformations in the opposite
solutions are available. For instance, there are various types of correlations other direction; that is, from ordinal or metric measures into categorical ones. For
than the product-moment coefficient; one of these, the point-biserial correlation instance, ordinal data can also be analysed, though with some loss of
coefficient, might be applicable to this example. This coefficient is designed to information, with the aid of the non-parametric techniques applicable to
express the relationship between a metric measure and a dichotomy. Our nominal data (percentages and Chi-squared tests). This latter approach is often
nominal variable - in this case, religion - could be regarded as a series of used for the sake of uniformity and simplicity, though it takes no account of the
dichotomies, that is Protestantlnon-Protestant; Roman Catholiclnon-Catholic; ordinal attributes of the data.
Muslimlnon-Muslim and so on. Each of these dichotomies could be related to It is also worth pointing out that interval measures can readily be transformed
our interval scale - in this case, alcohol consumption - and a series of point- into categorical data (albeit with some loss of information) by arranging them
biserial correlation coefficients could be calculated. (Warning: if we follow this into frequency distributions. Thus, instead of calculating the average age of a
procedure for a number of variables we shall greatly increase the number of sub-sample, we may say that 24 per cent were over forty-five years old. Since
correlation coefficients, to the point where some of them might be statistically this involves considerable loss of information, why would we do this? Suppose
significant purely by chance.) we have obtained age in terms of date of birth, and we wish to create a table in
O r we could approach the problem in another way, by calculating the average which age is related to another variable, a nominal one, for example to brand
or mean alcohol intake (assuming that we have an interval-type measure of this) preferences of a particular product. Imagine how such a table would look: across
for each religious group and then performing a one-way analysis of variance and the top we have, say, eight hfferent brands: Brand A, Brand B and so on,
an F-test on the data. This would tell us whether, in our sample, the average heading eight separate columns; and down the side we have age, calculated from
alcohol intake differs significantly according to the respondent's religion. each respondent's date of birth - running, say, from eighteen years three
Perhaps further analysis might show that one religious group drinks signifi- months down to sixty-five and over! Such a table would require many print-out
cantly more, or less, than most of the others. But whatever approach we use, we pages and would be quite unmanageable. Obviously, before asking for such a
cannot generate a single summary coefficient to describe the overall strength of table, we would transform the age variable into, say, half-a-dozen class intervals.
this association. This is the type of analytic and presentational problem that Now we have a table of eight columns and six rows, which is much more
arises in the case of 'mixed' categorical and interval scale data. intelligible, even though we have turned the metric variable 'age' into a much
To try to avoid some of these cumbersome procedures when we try to inter- cruder ordinal classification.
relate variables of different types, we sometimes engage in data transformation. In However, as we have seen, nominal data cannot be turned into ordinal or
our first example we showed how some data can be transformed (in that case interval ones unless we are willing to make certain assumptions; for example,
from nominal to dichotomous). Other transformations are possible. For about the nature of social class or about the response categories of attitude
example, groupings of jobs or occupations can be regarded as purely nominal items. When tempted to indulge in some data transformation of a dubious
categories of economic activity in no particular order, or they can be placed in a nature, it might therefore be useful to carry out the analysis several times, each
prestige hierarchy (see page 269) and used as a ranking of socioeconomic time using a different assumption. Such a procedure might well show that
background with perhaps half-a-dozen levels, that is they will have been making some radical data transformations is not really necessary and that
transformed into an ordinal scale. Some researchers would go farther, and turn almost equally clear results can be obtained by more conservative means.
these ranks into an interval scale, with numerical values such as 5 or 2 given to A final statistical comment refers to the exact nature of the question we want
every occupation. Researchers are driven to such dubious transformations by to ask of our data. We may, for example, find a statistically significant pattern of
the desire to apply parametric techniques of analysis (such as ANOVA, associations between certain variables, but this tells us little about either strength
correlations, and regression methods), since these are more powerful and more or direction (which 'causes' which) of the associations, nor anything about their
versatile than non-parametric methods such as the Chi-squared test. predictive powers. Thus, we may find a significant association (for example a Chi-
However, while such transformations may be useful, they require us to make squared test result which significantly departs from chance) between political
certain assumptions. Are we, for instance, willing to assume that occupations party allegiance and a measure of ethnic prejudice, but we cannot calculate a
can be reliably placed into a prestige hierarchy? A single hierarchy? In which coefficient to indicate its strength. Without doing much further research we do
culture? For women, too? And are we then willing to turn them into an interval not know its direction (which came first, choice of party or ethnic prejudice?);
288 Questionnaire Design, Interviewing and Attitude Measurement Statistical analysis 289
nor can we tell its power to make causal predictions of, say, voting behaviour. So question on attitudes to abortion, we might find a different result. Perhaps now
what exactly is the question to which we hope that our data analysis will provide we will find 53 per cent of our middle-class respondents giving a favourable
an answer? reply. Again, let us look at the working-class figures; quite likely we shall find
Furthermore, we must always remember that statistical significance does not them running at around 32 per cent in favour for our second question. In other
indicate social significance or practical usefulness. If our sample is large enough, words, it is often impossible to say, in absolute terms, how many respondents
a difference between average scores of 23.8 and 24.2 on an attitude scale may be have a particular attitude to a given extent, because so much depends on the
statistically significant, but is it meaningful, or socially relevant? wording of the question (see page 148). Nevertheless, the relative differences
(between classes, age groups, sexes or some other variable of concern) may well
be quite stable and consistent in their direction, no matter how the question is
Finally phrased. In social research we have few absolute measures, but relative
differences are well worth having if they are consistent. They constitute a 'real'
There is no end to statistical argumentations, but perhaps two more issues from finding and can give us an indication of relationships between underlying
among many may be singled out for discussion. variables.
The first is the problem of difirence scores or change scores. These arise in
longitudinal studies (for example, how stable are people's preferences, over time,
for product X or for certain types of music or health foods?) and in before-and-
Selected readings
after experiments (for example, the impact of a TV programme on certain
attitudes, or the effects of a new teaching method in schools). (For longitudinal STATISTICS
TEXTS
studies, see Chapter 2.) To create an oversimplified example, suppose that just Rossi, P.H., Wright, J.D. and Anderson, A.B. (eds), 1983, Handbook of Suvvey Research,
before an election we find that 40 per cent of our sample would vote for party Academic Press, New York.
P and that, when we go back to the same respondents six months later, we find See chapters 11-16.
again that 40 per cent express a voting intention for party P. Does this indicate Wright, George and Fowler, Chris, 1986, Investigative Design and Statistics, Penguin Books.
that there has been no change? Not necessarily for, theoretically, it is possible A practical and instructive basic text.
that every respondent has changed political allegiance and that none of those
who intended to vote for party P on the first occasion still intends to do so on the Zeisel, Hans, 1957, Say It With Figures, Harper, New York.
Still a very useful and approachable text, with good advice on cross-tabulations, on
next! And what if the figures had been, say, 35 per cent on the first occasion, and
Don't Know responses and on panel studies.
50 per cent on the next - does this prove that party P has broadened its support
by holding on to its original adherents while capturing an additional 15 per cent Howell, D.C., 1989, Fundamental Statistics for the Behavioural Sciences, 2nd edition, Chapman
of the vote? Again, not necessarily; it is theoretically possible that party P has lost & Hall, London.
all its original supporters and that its current adherents are all new recruits. Healy, J.F., second edition, 1990, Statistics: A Tool for Social Research, Chapman & Hall,
Thus, total-sample statistics can mask, or under-estimate, underlying change. London.
What is necessary is the calculation of a change score for each individual in the sample.
Skinner, C.J., Holt, D. and Smith, T.M.F., 1989, Analysis of Complex Suvveys, Wiley,
Where the dependent variable is more complex than simple party preference, Chichester, Sussex.
such calculations can become quite demanding; for example, studying shifts in For the advanced student.
readership of a daily newspaper in a city that has six. Moreover, the statistical
treatment of the results, especially if the variable is a categorical one, becomes a O'Muircheartaigh, C.A. and Payne, C., 1977, The Analysis of Survey Data, 2 vols, Wiley,
permutational ordeal; and there are problems, too, with the assessment of Chichester, Sussex.
More advanced methods of data exploration and model fitting.
reliability and of statistical significance. So it is as well to be aware of these
problems and, as far as possible, to anticipate them before undertaking
longitudinal research.
The second issue also has to do with comparisons, but not with comparisons
Neave, H.R. and Worthington, P.B., 1988, Distribution-free Tests, Unwin Hyman, London.
over time. The problem arises when we have used more than one technique to Introductory text on non-parametric methods, a successor to Siegel's book (see
measure the same attribute or variable. Suppose that we have asked a number below) of 1956.
of questions to measure attitude to abortion, and that on our first question we
have found 62 per cent of middle-class respondents to be 'in favour'. By itself, Siegel, S., 1956, Nonparametric Statistics for the Behavioural Sciences, McGraw-Hill, New York.
this figure is very nearly meaningless. It only acquires some meaning when we A widely used and very intelligible text.
can compare it, say, to a response of 48 per cent 'in favour' among working-class Dunteman, George H., 1989, Principal Components Analysis, Quantitative Applications in
respondents. But perhaps if we look at the results of a differently worded the Social Sciences No. 69, Sage, London.
290 Questionnaire Design, Interviewing and Affifude Measurement
Kalton, G., 1983, Compensating for Missing Survey Data, Institute for Social Research,
University of Michigan.
Careful treatment of the problems of missing cases and missing items, advising on
Appendix I
weightings, estimates and other possible adjustments.
Rubin, D.B., 1987, Multiple lrnpufafion for Nonresponse in Surveys, Wiley, New York.
For the specialist.
Himmelweit, Hilde T., Humphreys, Patrick, Jaeger, Marianne and Katz, Michael, 1981,
How Vofers Decide, Academic Press.
A sophisticated statistical treatment of causality.
Everitt, Brian S. and Dunn, Graham, 1991, Applied Multivariate Data Analysis, Edward
Arnold, Hodder & Stoughton, Sevenoaks, Kent.
Deals with problems of dimensionality, MDS, cluster analysis, regression, factor
analysis and discriminant analysis. For advanced students.
If you get more than one answer wrong, you should not embark on survey
research without a refresher course in statistics.
1. Imagine that you had asked a group of men and a group of women: 'What
kind of music do you like best of all?' How would you find out whether there is
a significant gender difference in the type of music preferred?
(a) Calculate the means and use the t-test to test for statistical significance;
(b) use the F-test to test for statistical significance;
(c) construct two frequency distributions and apply the Chi-squared test for
statistical significance;
(d) calculate the standard deviation for each gender and compare;
(e) none of the above.
Chart 1 Chart 2
298 Appendix II
48-50; organization of, 50-51; processing questionnaire(s), 10;bias in, 102-3, 106; analysis of, 27-8; piloting of, 49, 60-61, sub-groups, 44,283-4
data from, 61-2; on questionnaires, 47-8, group-administered, 103,141; increasing 62-3; random, 40, 98; representative, 8, 38, supplementary (probe) questions, 53, 74,
59, 60-61; on questions, 49, 52-6, 58, 62-3, response rates to, 103-6; layout of, 59-60, 39-42; size of, 43-4; snowballing, 43 90-91,113,138
117, 125, 130; report on, 63-4; sample for, 105, 120, 122, 130; mail, 48, 60, 102-3, 105, sampling frames, 8, 39, 42, 98 survey design: analytic, 11, 12-13, 21-36;
49,50-61,62-3; on scaling, 63 122; non-response to, 106-7; pilot work on, scaling: attitude see attitude scales; errors in, appropriate, choice of, 35-6,44-5; case
placebo effect, 30 47-8,59, 60-61; question modules in, 159,165-6; linear model of, 151-5,165, study of, 10-11; compared with
play techniques, 227-8 108-10; question order in, 110-12; school 187-8; ordinal, 155-6, 158, 285; pilot work experiments, 11-122, 16; descriptive, 12-13,
politeness, 122, 212 children and, 107-8,141; self-administered, on, 63; 'quick' methods of, 229-59; reliability 38-45; first steps in, 7-10; need for, 5-7;
populations, 38 103,122,141; specification of, 100-101; of, 159-60, 162, 188, 283; validity of, 153, problems in, 13-17,18; testing of adequacy
postal surveys see mail surveys type of, 102-3 160-63,188,205-8 of, 9-110,49-50
post-enumeration surveys, 56 quota sampling, 41, 96 scalogram analysis, 201-5 survey techniques, 6, 8
Postlethwaite, T. N., 236 scapegoating, 17 systematic errors, 87
predictive validity, 160, 161-2, 206 random allocation procedures, 31 schoolchildren see children
preparatory work see pilot work random digit dialling, 98 scientific experiments, 11-12, 16 tape recording of interviews, 52,67, 71
prestige (social desirability) bias, 109, 127, 133, random errors, 87 scientific method, 13, 17 Tau correlation coefficient, 250, 285
138-40,181 random sampling, 40,98 scoring systems, 163; for projective technical terms, avoidance of, 129
pre-testlpost-test survey designs, 29 range checks, 279 techniques, 216, 218-19 telephone interviewing, 97-9
prizes, 104 ranking scales, 155-6,158,250-52 search process, mental, 85-6 temporal sequence, causality and, 15
probability sample, 8, 39 rapport, maintenance of, 73, 82, 89-90,105 selection, explanation of reasons for, 104 test-retest reliability, 160
probes, 53,74,90-91,113,138 rating scales, 230-36 self-administered questionnaires, 103, 122 141 tests, psychological see projective measurement
problem respondents, 94 recruitment: for depth interviews, 68-9 self-incrimination barrier, 211-12 techniques
projective measurement techniques, 156, recurrent behaviour, questions on, 130-32 self-selection bias, 30-31 Thematic Apperception Test, 220
210-14; cartoons, 219-20; picture regression analysis, 27-9 semantic differential, 236-9, 287 Thurstone, L. L., 190
interpretation, 220-23; play as, 227-9; relational (analytic) survey design, 11, 12-13, sensitive topics, 140-41 Thurstone scales, 190-95
pseudofactual questions and, 225-7; 21-36 sentence completion technique, 56, 179, Torney, Judith V., 184,245
sentence-completion, 56, 179, 214-18; relevance, problem of, 117 214-18 training: of interviewers, 82, 87, 142
stories and, 224-5 reliability: of attitude questions, 147-8; of significance, statistical, 287-8, 294 transformation of data, 286-7
projective questions, 58, 71, 74 attitude scales, 163-6; of coding, 2 6 3 8; of silence: in interviewing, 74 translation problems, 184,201
prompts, 90 questions, 144-8; of scaled measures, simple random sample, 40 tree model, 176
prospective longitudinal surveys, 34 159-60,162,188,283 situational problems in interviews, 94 typology, development of, 28
proverbs and sayings, avoidance of, 128 reminders, 105 Snow, John, 15-16
pseudofactual questions, 225-7 repertory grid, 239-41 snowballing samples, 43 unanswered questions, 119-20
psychological tests see projective measurement repetition, use of, in interviewing, 75 social desirability (prestige) bias, 127, 138-40, uncontrolled variables, 21-2
techniques report writing, 8; and pilot work, 49, 63-4 181 undersampling, 42, 96
psychometrics, 144 representative sample, 8, 38, 39-42 social-distance scales, 189-90 univariate analysis, 281-2
publicity, 104 reproductibility, 188 sociodemographic checks, 103,106 unstructured interviews, 51-2
research design see survey design sociometry, 254-9
quality checks, 146 research techniques, 6, 8 Spearman's Rho correlation coefficient, 250, validity: of attitude questions, 147-8; of
question(s): attitude, 87, 88-9, 113, 143-4, response(s); anonymity of, 49,105, 265; to 285 attitude scaling, 205-6; concurrent, 160,
147-8,150; awareness, 109, 247; bias from attitude statements, 179-80, 181-5; barriers special procedures in interviews, 94-5 161, 162; content, 162; face, 161; predictive,
poor design of, 53,121; check, 92; to, 211-12; confidentiality of, 104-5, 108, spiral reinforcement process, 18 160,161-2,206; of questions, 144-8; of
classification, 87, 108-9, 132-7, 139; closed, 265; increasing rates of, 103-6; instructions split ballots, 61, 125, 130, 145 scaled measures, 153, 160-63,188, 205-8
112,113-14,117,127,129; double-barrelled for, 141-3; motivation of, 82; piloting of split-half method, 160 variable(s), 7, 8, 101; types of, 21-4
and double-negative, 126, 128; factual, 52-6, categories of, 53; problem, 94; process of, sponsorship, 104 variable allocation document, 263
87, 88, 109, 124-7, 145-7; filter, 111-12; 85-6,121; see also coding of responses; standardized interviews, 65, 66-7, 81-99, volunteer bias, 30-31
free-response, 112-13,114; funnelling, statistical analysis 142-3
110-11; hypothetical, 126; in interviews, retrospective longitudinal surveys, 34 statistical analysis, 8, 157-9, 281-9; test on, women: as interviewers, 84
71-2,73-4,87-9,90-91,92; knowledge, 94, return envelopes, 48, 105 291-3 wording of questions, 76, 119-27; rules for,
139, 247; leading, 53, 74, 75, 129-30, 137-8; reverse order testing, 9-10,49-50 statistical weighting, 106 128-30; special problems in, 130-41,143-8
modules of, 108-12; multiple-choice, 52-3, rewards (incentives), 82,104 stimulus equivalence, 86, 87, 121 words see language
114,117, 138; open-ended, 49, 52, 74, Rho correlation coefficient, 250, 285 story technique, 224-5
112-13,114,127,129; order of, 110-12; role reversal, 72 students, piloting of questionnaires on, 62 Zubin, Joseph, 294
personal data, 108-9; piloting of, 49, 52-6, routing instructions, 92
58,62-3,117,125,130; probes, 53,74, running prompts, 90
90-91,113,138; pseudofactual, 225-7; sample(s): design of, 8; drawing of, 8, 39-42;
purpose of, 53,120, 123-4; reliability and enumeration, 26, 33; errors in, 43-4; follow-
validity of, 144-8; wording of, 76, 119-41, up, 8; interviewing, 96; judgement, 43;
143-8; see also response(s) matching of, 32-3; multiple regression