Designing and Conducting Survey Research: A Comprehensive Guide

PART ONE

DEVELOPING AND ADMINISTERING QUESTIONNAIRES

CHAPTER ONE

AN OVERVIEW OF THE SAMPLE SURVEY PROCESS

Surveys have become a widely used and acknowledged research tool in most of the developed countries of the world. Through reports presented by newspapers, magazines, television, and radio, the concept of considering information derived from a relatively small number of people to be an accurate representation of a significantly larger number of people has become a familiar one. Surveys have broad appeal, particularly in democratic cultures, because they are perceived as a reflection of the attitudes, preferences, and opinions of the very people from whom the society's policymakers derive their mandate. Politicians rely heavily on surveys and public opinion polls for popular guidance in mapping out campaign strategies and carrying out their professional responsibilities. Commercial enterprises use survey findings to formulate market strategies for the potential widespread use, distribution, and performance of new and existing products. Television and radio programs are evaluated and scheduled largely in accordance with the results of consumer surveys. Government programs designed to provide assistance to various communities often rely on the results of surveys to determine program effectiveness. Private social organizations obtain information from their members through the use of survey techniques. Libraries, restaurants, financial institutions, recreational facilities, and churches, among many others, make use of polls to solicit information from their constituents and clientele concerning desired services.

As a research technique in the social sciences and professional disciplines, survey research has derived considerable credibility from its widespread acceptance and use in academic institutions. Many universities have established survey research institutes where the techniques of survey research are taught and surveys can be conducted within the confines of propriety and scientific rigor. Students are often encouraged to use survey research for gathering primary data, thereby satisfying the requirement of conducting original research. Professors publish countless articles and books based on the results of funded and unfunded survey research projects.

Despite the broad-based societal acceptance of survey research, there remains a lingering doubt, especially within the general population at large, concerning the reliability of information derived from relatively few respondents purporting to represent the whole. They frequently ask, for instance, How can fifteen hundred respondents to a survey be said to represent millions of people? or, Why should two thousand television viewers dictate to program directors on a national scale what Americans choose to watch? The answers to these and other such questions lie in the systematic application of the technique of scientific sample survey research.

Survey research involves soliciting self-reported verbal information from people about themselves. The ultimate goal of sample survey research is to allow researchers to generalize about a large population by studying only a small portion of that population. Accurate generalization derives from applying the set of orderly procedures that comprise scientific sample survey research. These procedures specify what information is to be obtained, how it will be collected, from whom it will be solicited, and how it will be analyzed.

If the researcher needs personal, self-reported information that is not available elsewhere and if generalization of findings to a larger population is desired, sample survey research is the most appropriate method as long as enough general information is known or can conveniently be obtained about the subject matter under investigation to formulate specific questions and as long as the population that is needed to be sampled is accessible and willing to provide self-reported information. The theoretical underpinnings of scientific sample survey research, its procedural applications, and analysis of the data it generates constitute the substance of this book.

Gathering Information through Research

Researchers must be aware that survey research is only one among several methods associated with the process of data collection. The three main techniques used to collect primary data (data collected firsthand, directly from the subjects under study) are survey research, direct measurement, and observation, all of which in one way or another can make use of sampling. Secondary research is a fourth means of data collection. It consists of compiling and analyzing data that have already been collected and exist in usable form. These alternative techniques, when they are not appropriate in and of themselves, can often be used as complements to the survey research process. A brief description of these alternative techniques follows:

Secondary research: Certain data may already exist that can serve to satisfy the research requirements of a particular study. Any study should investigate existing sources of information as a first step in the research process to take advantage of information that has already been collected and may shed light on the study. Sources of secondary information include libraries, government agencies, and private foundations, among others.

Direct measurement: This technique involves testing subjects or otherwise directly counting or measuring data. Testing cholesterol levels, monitoring airport noise levels, measuring the height of a building to make certain it complies with local ordinances, and counting ballots in a local election are all examples of direct measurement.

Observation: A primary characteristic of observation is that it involves the direct study of behavior by simply watching the subjects of the study without intruding on them and recording certain critical natural responses to their environment. For example, a government official can obtain important information about the issues discussed in a speech by observing the audience's reactions to that speech.

However, there is no better method of research than the sample survey process for determining, with a known level of accuracy, detailed and personal information about large populations. Opinions, which are the keys to public policy, are obtainable with defined and determinable reliability only through the survey research process. By combining surveys with scientific sampling, the researcher is using the only method of gaining this information to a known level of accuracy. The survey process is particularly suited to collecting data that can inform the researcher about research questions such as the following:

How do Americans feel about proposed changes in social security regulations?

What is the average income of people twenty-five years of age and older whose highest level of completed education is high school?

What factors influence people's choice of banks?

What are the reactions among employees of a local factory concerning a newly proposed union policy?

How do members of the New York State Bar Association feel about capital punishment?

What do various state legislators think about a proposed mandatory balanced-budget amendment?

What proportion of drivers observe seat belt laws?

To what extent has the Latino community in Texas experienced job discrimination?

The particular use for which a survey is conducted determines the informational requirements of that survey. Surveys typically collect three types of information: descriptive, behavioral, and attitudinal.

Surveys frequently include questions designed to elicit descriptive information or facts about the respondent. Such important data as the respondent's income, age, education, ethnicity, household size, and family composition are integral to most sample survey studies. These socioeconomic characteristics provide important information that enables the researcher to better understand the larger population represented by the sample.

In many survey research projects, the researcher is interested in the respondent's behavior. Patterns of transportation use, recreation, entertainment, and personal behavior are often the desired information in sample survey studies. For example, such information as frequency of public transit ridership or use of various types of recreational and entertainment facilities is typical of behaviorally oriented information that can be obtained from sample surveys.

In addition to descriptive and behavioral information, many surveys solicit, as their primary focus, the respondent's attitudes and opinions about a variety of conditions and circumstances. The hallmark of this type of sample survey is the public opinion poll, which seeks opinions and preferences regarding issues of social and political relevance. The primary objective of such studies is to be predictive and future oriented.

Very rarely does a study include only one of the informational categories we have noted. Scientific investigation requires that relationships be identified in terms of descriptive, behavioral, and attitudinal data so that we may fully understand the differential complexities of the population from which a sample has been drawn. For instance, in a political public opinion poll, it is much more desirable to know not only the breakdown of votes for each candidate but also such factors as the voter's political party, age, gender, past voting patterns, and opinion on a variety of key issues. Such a survey requires the researcher to derive information from each of the above categories in one sample survey.

Advantages of Sample Survey Research

Generalizations based on a mere fraction of the total population (a sample) did not gain acceptance until the beginning of the twentieth century, when a researcher for a liquor distillery in England named W. S. Gossett was faced with the problem of testing the quality of his company's product. Testing the plant's output involved tasting, and therefore consuming, the product. Testing the entire output of the plant, or even as few as one in ten bottles, was clearly not economically feasible. Gossett, writing under the pseudonym Student, developed a theoretical basis for making generalizations about the quality of the plant's product by sampling only a small portion of that output.

The foremost advantage of the sample survey technique, as indicated by Gossett's experience, is the ability to generalize about an entire population by drawing inferences based on data drawn from a small portion of that population. The cost and time requirements of conducting a sample survey are significantly less than those involved with canvassing the entire population. When implemented properly, the sample survey is a reasonably accurate method of collecting data. It offers an opportunity to reveal the characteristics of institutions and communities by studying individuals and other components of those communities that represent these entities in a relatively unbiased and scientifically rigorous manner.

Surveys can be implemented in a timely fashion. That is, the survey project can be organized so that the actual data gathering is performed in a relatively short period of time. Besides the convenience afforded by this approach, there is also the advantage of obtaining a snapshot of the population. Other techniques may involve a longer-term study, during which opinions or facts may change from the beginning of the study to the end.

Well-structured sample surveys generate standardized data that are extremely amenable to quantification and consequent computerization and statistical analysis. This quality has been enhanced through rapid advances in computer technology as well as through the development and refinement of complex analytical statistical software packages and techniques. For purposes of comparisons among individuals, institutions, or communities, surveys offer a further advantage: replicability. A questionnaire that has been used in one city or community can be reimplemented in another community or administered once again in the same community at a later date in order to assess differences attributable to location or time.

Sampling started gaining general acceptance beginning in 1935 when George Gallup established the American Institute of Public Opinion in order to conduct weekly polls on national political and consumer issues for private and public sector clients. In as much as Gallup was operating a business for profit and since he was committed to delivering weekly polls, he was necessarily highly sensitive to cost and time factors. Gallup developed a method of sampling fifteen hundred to three thousand respondents—quite a small number compared to other surveys at that time. His method established sample quotas based on age, sex, and geographical region. In the 1936 presidential election between Franklin D. Roosevelt and Alfred Landon, Gallup forecast a Roosevelt victory, while one of the most respected polls at that time, the Literary Digest poll of 2.5 million subscribers, forecast a Landon landslide. The final results are well known: a Roosevelt victory with 61 percent of the vote. The scientifically implemented small sample thereafter became established as the survey method of choice. Advancements in the understanding of sample survey methodology that were developed in World War II and refined thereafter now provide even greater accuracy than Gallup had in 1936, with still smaller sample sizes.

Types of Sample Survey Research

Survey information can be collected by means of any of five general methods of implementation: mail-out, web-based, telephone, in-person interviews, and intercept. This section addresses the advantages and disadvantages of these types of surveys and discusses the procedures for administering the surveys.

Mail-Out Surveys

The mail-out format for collecting survey data involves the dissemination of printed questionnaires through the mail (commonly the postal service) to a sample of predesignated potential respondents. Respondents are asked to complete the questionnaire on their own and return it by mail to the researcher (postage paid by the researcher).

Advantages and Disadvantages.

The advantages of the mail-out technique can be stated as follows:

Possible cost savings: Other techniques require trained interviewers, and the recruitment, training, and employment of interviewers can be quite costly. Access to respondents by mail can, under some circumstances, be less expensive than telephone surveys and certainly less expensive than in-person interviews.

Convenience: The questionnaire can be completed at the respondent's convenience.

Ample time: The respondent has virtually no time constraints. There is enough time to elaborate on answers and consult personal records if necessary to complete certain questions.

Authoritative impressions: The researcher can prepare the mail-out questionnaire form so that it has significant legitimacy and credibility.

Anonymity: Because there is no personal contact with an interviewer, the respondent may feel that the responses given are more anonymous than is the case with other formats.

Reduced interviewer-induced bias: The mail-out questionnaire exposes all respondents to precisely the same wording on questions. Thus, it is not subject to interviewer-induced bias in terms of voice inflection, misreading of the questions, or other clerical or administrative errors.

Complexity: Mail-out questions can be longer and more complex than telephone questions.

Visual aids: Mail-out questionnaires can make use of photographs and maps that would be impossible to use in a telephone survey.

Mail-out questionnaires, however, have the following disadvantages:

Comparatively long time period: Many follow-ups and substitutions of sample respondents are required in order to achieve the appropriate sample size and adequate random distribution necessary for purposes of generalization. The mail-out therefore generally requires a few weeks for questionnaires to be returned.

Self-selection: Mail-outs typically achieve a lower response rate than telephone surveys. Low response rates can imply some bias in the sample. For instance, poorly educated respondents or those with reading or language deficiencies tend to exclude themselves from this form of survey more often than from surveys administered by an interviewer.

Lack of interviewer involvement: The fact that no interviewer is present means that unclear questions cannot be explained, there is no certainty that the questions will be answered in the order written (which may be important), and spontaneously volunteered reactions and information are not likely to be recorded by the respondent and cannot be probed by an interviewer as would be the case with other methods.

Incomplete open-ended questions: It is more likely that questions requiring an original written response in lieu of fixed answers will be avoided.

Administration of Mail-Out Surveys.

Certain guidelines should be followed in administering a mail-out questionnaire. First, the questionnaire should be designed in the form of a booklet in order to ensure a professional appearance and to make it more usable by the respondent. Any resemblance to an advertising brochure should be strictly avoided. The aesthetic appearance of the questionnaire is important in terms of generating satisfactory response rates. There should be adequate spacing between questions, and questions should not begin on one page and end on another. Instructions to the respondent should be clear and easily distinguished from the survey questions themselves. Graphics, such as maps and illustrative photographs, should be carefully integrated into the design of the questionnaire.

The cover letter should be prepared in accordance with the principles discussed in chapter 2 and should become the first page of the booklet. The last page of the booklet should be reserved for three purposes only: to express appreciation to the respondents for their participation, provide a return mailing address and prepaid postage through a business reply permit, and provide instructions for returning the completed questionnaire. An alternative is to provide postage-paid, preaddressed return envelopes, but the cost of this approach is somewhat higher.

Questionnaires should be stamped with an identification number or bar code for purposes of monitoring the follow-up process. This number or code must be explained to the respondent in the cover letter, accompanied by assurances of privacy and confidentiality.

The questionnaire booklet is mailed by first-class postage to the respondent in an envelope. The envelope is addressed with the name and address of the respondent individually imprinted (in the case of small, more personalized surveys) or with a mailing label (most commonly used in large-scale, high-volume surveys). A target date should be designated for the return of the questionnaire; this target date is generally recommended to be approximately two weeks from the initial mailing date. Two weeks after the initial mailing, a follow-up postcard reminder should be sent to potential respondents who have not yet replied, as determined by their prestamped identification number. The reminder should be friendly in tone and indicate that if the completed questionnaire and the reminder postcard have crossed in the mail, the respondent should disregard the reminder; it should also again express appreciation for the respondent's cooperation.

Four weeks from the initial mailing, a second follow-up can be mailed to all survey recipients who have not yet responded. This follow-up should include a new cover letter that does not specify a target due date but instead stresses the importance of responding. Another copy of the questionnaire should accompany the letter in case the original questionnaire has been misplaced or discarded.

It can be reasonably expected that this procedure will yield a response rate that can approach 50 percent for the general public and a somewhat higher rate for specialized populations. The researcher should wait two weeks after the second follow-up before closing the mailing process. A response rate of 50 percent can be considered satisfactory for purposes of analysis and reporting of findings as long as the researcher is satisfied in the representativeness of the respondents (see chapter 9). If the researcher wishes to increase the response rate and has adequate resources and time to do so, the following additional procedures are suggested:

In lieu of using mailing labels, envelopes and the cover letter can be individually imprinted with the potential respondent's name and address.

The cover letters should be individually signed in blue ink to avoid the impression that they were impersonally mass-produced.

The follow-up mailings should include eye-catching but tasteful illustrations and graphics.

Six weeks after the initial mailing, nonrespondents can be given a reminder telephone call.

A third follow-up mailing, again with a new cover letter and copy of the questionnaire, can be sent to all nonrespondents eight weeks after the first mailing. This third follow-up should be delivered by certified mail.

These additional procedures are designed to achieve a response rate higher than 70 percent for the general population and as high as 90 percent for certain specialized groups.

Web-Based Surveys

The web-based survey is an alternative to the traditional mail-out technique; individuals are contacted by e-mail and asked to participate in a survey that is designed to be completed and submitted through the Internet.

Advantages and Disadvantages.

The advantages of the web-based survey method are as follows:

Convenience: This technique represents a convenient and efficient way of reaching potential respondents. They are able to receive the questionnaire and complete it in the privacy of their home or office. This advantage is becoming particularly significant as the availability of computers becomes increasingly widespread.

Rapid data collection: Information, especially information that must be timely (e.g., a political public opinion poll related to an upcoming election), can be collected and processed within days.

Cost-effectiveness: This technique is more cost-effective than the traditional mail-out survey because there is no need for postage or paper supplies. It is also more cost-effective than the telephone and in-person surveys because it is not at all labor intensive.

Ample time: The respondent is not pressed for time in responding to the web-based survey and has the opportunity to consult records in answering the questions. There is time to consider response choices and respond to open-ended questions in the form of text.

Ease of follow-up: Potential respondents can be reminded to respond to the survey through follow-up e-mail messages.

Confidentiality and security: Personal or sensitive information supplied by the respondents can be protected on a secure server through the efforts of the research team.

Specialized populations: The survey is particularly useful in reaching specialized or well-identified populations whose e-mail addresses are readily available. For example, we have successfully used this technique to conduct surveys of satisfaction among employees and stakeholders of large public organizations.

Complexity and visual aids: As with mail-out surveys, web-based surveys can use visual images and more complex questions.

Web-based surveys also have certain disadvantages:

Limited respondent bases: A major disadvantage of this technique is that it is limited to populations that have access to e-mail and a computer. Furthermore, the technique assumes a certain minimal level of computer literacy that is necessary for the completion and submission of the questionnaire. Such literacy is improving rapidly within the general population.

Self-selection: As in the traditional mail-out, there is a self-selection bias that leads to lower response rates. Those who do not use e-mail or are not comfortable with web-based technology exclude themselves from the sample. Also, individuals with reading or language issues tend not to respond to web-based surveys. Some researchers send the survey by e-mail in multiple languages in an effort to obviate this problem.

Lack of interviewer involvement: Since there is no interviewer involvement in the web-based survey, unclear questions cannot be explained, and respondents may not follow instructions. These problems can seriously compromise the scientific reliability of the survey even though telephone contacts are provided to the respondents in the event that they need help.

Administration of Web-Based Surveys.

The web-based survey requires expertise to prepare a survey instrument for online administration. If the research team members have the necessary expertise and the necessary equipment, including a secure server, they may set up the survey for online response themselves. Alternatively, the researcher can use a service that can be purchased, such as Survey Monkey, where the instructions for setting up the survey structure, sending it to the intended recipients, and receiving responses are explicit and relatively straightforward. If such a service is used, it is possible to convert the data file to the Statistical Package for the Social Sciences (SPSS) for a more robust analysis. The following administrative tasks are required to implement a web-based survey successfully:

The researcher must have the capability of providing a survey Internet link to the intended respondents. For example, respondents may be employees in a company who are asked to complete an online job satisfaction survey. The researcher will be given access to the e-mail addresses of these respondents in order to send them the survey link. The researcher should be able to send e-mail messages to everyone simultaneously as well as to other selected individuals as desired and necessary. In other types of research efforts, the researcher may be interested in a survey with a broader base of potential respondents—perhaps extending to an entire community. In this case, the researcher can advertise in local newspapers and through community organizations that the survey is available by visiting a specific website and clicking on the appropriate survey link.

The researcher should strive to prepare an online questionnaire that is as user friendly as possible. The online program should guide the respondent efficiently through the questions to the final submission of the completed survey.

The initial e-mail message or the introduction to the online survey itself should specify a deadline for the return of the survey form—about ten days. If the researcher knows the e-mail addresses of those who received the survey link, approximately five days after sending the initial e-mail, the researcher may consider sending a reminder e-mail to those who have not yet responded.

It is important that the questionnaire be submitted to a secure server so that the privacy of the respondents is maintained.

Web Panels.

Web panels are large reservoirs or banks of potential survey respondents who are recruited to participate in various online surveys. Panel or web companies generally recruit potential respondents to join the web panel through advertisements on the Internet and in local newspapers and newsletters. Web companies encourage panel participation through a series of incentives, including an opportunity to earn money, a chance to have a voice in new products and services, and an avenue to enjoy a variety of interesting surveys and thereby become introduced to various subject areas.

Potential panel recruits are directed to the company's website, where they are asked to provide varying amounts of personal and demographic information. This information is used to screen respondents for participation in surveys requiring a specific demographic profile. For example, a researcher may wish to purchase from the web panel administrator the e-mail addresses of panel members who are women between the ages of twenty and forty and who live in the New England states. Large web panels consist of 2 million to 3 million people. Like all other Internet surveys, those using a web panel have two essential disadvantages: (1) participant self-selection negates random selection and (2) individuals who do not have access to the Internet cannot join a web panel. The absence of such individuals can create a systematic bias in the sample population.1

The Telephone Survey

The telephone survey collects information through the use of telephone interviews between a trained interviewer and selected respondents.

Advantages and Disadvantages.

The advantages of the telephone survey interviewing process can be stated as follows:

Rapid data collection: Information, especially information that must be timely (e.g., a political public opinion poll related to an upcoming election), can be collected and processed within days. It is possible to complete a telephone survey in the time it would take simply to plan a mail-out or in-person survey.

Possible cost savings: The cost of implementing a telephone survey is considerably less than that of in-person interviews; under certain circumstances, it can even be less than that of a mail-out survey.

Anonymity: A telephone survey is more anonymous than an in-person interview. Hence, the interviewer can conduct in-depth questioning in a less-threatening environment than exists in face-to-face situations.

Assurance that instructions are followed: As with the in-person interview, the telephone interviewer can make certain that the questions are answered in precisely the order intended so that the integrity of the questionnaire sequence is maintained.

Telephone surveys also have certain disadvantages:

Less control: The interviewer has less control over the interview situation in a telephone survey than in an in-person interview. The respondent can easily end the interview at any time by hanging up the telephone.

Less credibility: The interviewer will have greater difficulty establishing credibility and trust with a respondent over the telephone than would be the case in person or by mail.

Lack of visual materials: Unlike the mail-out survey and the in-person interview, the telephone survey does not permit the use of visual aids, such as maps, pictures, or charts, as components of the questions.

Less complexity: Related to the lack of visual materials is the fact that telephone questions must be much less complex than most other survey forms.

Administration of Telephone Survey.

The telephone survey is less complex to implement than the mail-out. The most important aspect of this survey technique is the use of personal interviewers; the proper selection and training of these interviewers is critical to the success of the research project.

Selection of Telephone Interviewers.

Individuals may be recruited from a number of sources to serve as telephone interviewers. The single best source of interviewers, when available, is a local university. Students, especially upper-division undergraduate students and graduate students, are motivated to become involved in the interviewing process for two basic reasons. First, there is frequently some substantive interest in the research project and its potential findings. Second, students often seek ways to augment their income to help fund their education while at the same time gaining relevant experience and therefore may be willing to work for wages that are relatively modest in relation to their skill level. If the researcher does not already have an affiliation with a university, professors in appropriate disciplines should be contacted and arrangements made to recruit potential interviewers. University bulletin boards and newsletters can also be used. If universities are not easily available, temporary