Maximizing Your Number of Respondents Effectively and Ethically
Student participation is critical to survey success. Simply put, more respondents generally yield better quality data (Laguilles, Williams, & Saunders, 2011; Schaefer & Dillman, 1998). Sampling error, total completions, representativeness, and response rates are important measures of data quality (Cook, Heath, & Thompson, 2000), and are important for increasing confidence in the results among a wider audience. Despite the challenges of obtaining high response rates (Laguilles, Williams, & Saunders, 2011; Umbach, 2004), NSSE works with institutions to personalize materials used to contact each student, emphasizing the survey's value for institutional improvement. Institutional efforts can effectively supplement the invitation messages sent to students by NSSE. Survey participation is more likely when students are not suffering from survey fatigue. Therefore, institutions are advised to plan survey cycles to avoid contributing to survey fatigue. However, always keep in mind that students' decisions not to participate must be respected.
Each institution needs to assess their campus culture and determine appropriate methods to reach their students; there is no one correct way to increase student participation. Dillman (2007) indicates several important factors that contribute to higher response rates:
Perceived importance of the survey (value to the student, perceived legitimacy).
Level of interest students have in the topic
Creation of respondent trust
Increasing perception of rewards for participation
Decreasing perceptions of respondent burden and survey fatigue
The abovementioned factors are worth thinking about and addressing in survey promotion. Of course, survey publicity such as flyers and media articles as well as incentives provided for participation can help send a message to the whole campus that the data are valuable for institutional improvement. We outline suggestions for increasing response rates below, as well as practices that should not be used because they can result in undue influence on participation.
Including small incentives and lotteries* and incentives in contact materials (Laguilles, Williams, & Saunders, 2011)
Flyers, press releases, and videos
Informing students that survey responses will be kept confidential (Dillman, Singer, Clark, and Treat, 1996)
Multiple reminders with the sample (Dillman, 2007; Schaefer & Dillman, 1998)
Empirical research using college students and other populations indicate that the effectiveness of lotteries and incentives is mixed (Umbach, 2004). However, some NSSE schools do report that incentives appear to encourage student response, and recently published controlled experimental research suggests the effectiveness of incentives with a college student population (Laguilles, Williams & Saunders, 2011).
Be sure to include the estimated chance of winning a lottery on any promotional information. Institutions should state the expected number of respondents along with the quantity and value of prizes offered (e.g., “There will be drawings for two iPod Nanos ($149 value) and two $100 bookstore gift cards. About 400 students will be eligible to win one of these prizes.”).
More than five direct requests to participate in the survey
Direct and personal contact by campus officials with individual students
Revoking rights or privileges for non-response (e.g., blocking course or housing registration)
Improperly describing NSSE as an anonymous survey. Identified student responses are provided to institutional personnel, so survey responses are not anonymous
To help you develop your own NSSE promotional materials for your campus, we have assembled examples of how colleges and universities have promoted the NSSE survey on campus. These examples include ideas for encouraging students to participate in the survey, ways schools have disseminated their NSSE results, and recruitment materials.
Offer small incentives (e.g., bookstore gift certificates or tokens for free goods or services) for each respondent. These work best if the incentive is received in advance. For example you could send a voucher by mail or by postcard that can be "cashed in" after completing the survey. These can be difficult to administer, so often institutions choose lotteries or drawing.
Enter respondents in a drawing, noting the odds of winning (e.g., the number of students invited to complete the survey and number of prizes offered) and the value of the prizes in promotional materials. Lottery prize ideas include parking permits, institution memorabilia, or tickets to athletic events.
Send a press release to student and local newspaper.
Submit an article, or place an ad or insert in the student newspaper.
Submit an editorial to the student newspaper written by a student or administrator, explaining why the survey is important, the purpose, how the survey will be used and that the survey is sponsored by the institution; emphasize that students' participation is voluntary.
Post notices on the institutional Web site or course management systems.
Create a Facebook group or Twitter account and share results and update students
Signs on campus buses or along local transportation routes.
Produce locations for campus TV and radio stations.
Set up tables in the student center or union with survey information.
For additional ideas of how to encourage survey participation, click here
NSSE is both an institutional improvement effort and a research project, and as such, student participation in NSSE must be fully voluntary. Participants in research studies should always be informed of their rights relative to their participation and should know any potential risks of involvement in the study. NSSE provides this information in its contacts with students (see the informed consent statement). Additional efforts you make to increase response rates should never cause students to feel they will be penalized for not participating.
The Belmont Report (1979) established guidelines that led to the creation of Institutional Review Boards (IRB) or Internal Review Boards (IRB) to regulate research involving human subjects. This report outlined three considerations for beneficence in research: (a) maximize possible benefits, (b) minimize possible harms, and (c) equitably distribute the risks and rewards of research.
Inappropriate efforts to increase participation in research fall into two general categories: coercion and undue influence. Coercive interactions are those that imply directly or indirectly that a potential participant might lose rights or privileges for not participating in the study. Explicit examples of coercive measures include requiring students to complete the survey in order to register for classes or graduate. Implicit examples include the use of language such as “you must complete this survey” or “students with real NSSEville State University pride would complete this survey.”
Undue influence occurs when the incentives used to increase participation become the primary reason why students participate, or when the number or nature of attempts to encourage participation are excessive. Students should not perceive that the compensation is so great that they participate because they need the incentive, regardless of a sense that completing the survey would be a heavy burden to them.
Small incentives provided to each student who completes the survey are generally considered ideal. Larger incentives such as lotteries that draw prize winners from all participants may also be acceptable, but publicity for these efforts should include an estimate of the odds of winning. Survey promotion for these incentives should not emphasize the prizes to a degree that minimizes the requirement of survey participation.
References
Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web or internet-based surveys. Educational and Psychological Measurement, 60 (6), 821-836.
Dillman, D. A. (2007). Mail and Internet Surveys: The Tailored Design Method (2nd ed.). Hoboken, NJ: John Wiley & Sons.
Dillman, D. A. Singer, E., Clark, J. R., & Treat, J. B. (1996). Effects of benefits appeals, mandatory appeals, and variations in statements of confidentiality on completion rates for census questionnaires. Public Opinion Quarterly, 60, 376-389.
Laguilles, J. S., Williams, E. A., & Saunders, D. B. (2011). Can lottery incentives boost web survey response rates?: Findings from four experiments? Research in Higher Education, 52,( 537-553.
National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1979, April 18). The Belmont Report: Ethical principles and guidelines for the protection of human subject of research.( Washington, DC: Department of Health, Education, and Welfare. http://www.hhs.gov/ohrp/humansubjects/guidance/belmont.html
Schaefer, D. R., & Dillman, D. A. (1998). Development of standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62((3), 378-397.
Umbach, P. D. (2004). Web surveys: Best practices. In S. R. Porter (Ed.), Overcoming survey research problems ((pp. 23–38). Jossey-Bass: San Francisco.
Maximizing Your Number of Respondents
Effectively and Ethically
Student participation is critical to survey success. Simply put, more respondents generally yield better quality data. Sampling error, total completions, and response rates are important measures of data quality, and are important for increasing confidence in the results among a wider audience. NSSE works with institutions to personalize materials used to contact each student, emphasizing the survey’s value for institutional improvement. Institutional efforts can effectively supplement the messages sent to students by NSSE. However, always keep in mind that students’ decisions not to participate must be respected.
Each institution needs to assess their campus culture and determine appropriate methods to reach their students; there is no one “right way� to increase student participation. Survey research expert Dillman (2007) indicates several important factors that contribute to higher response rates:
Perceived importance of the survey (value to the student, perceived legitimacy).
Level of interest students have in the topic.
Creation of respondent trust.
Increasing perception of rewards for participation.
Decreasing perceptions of respondent burden.
Survey publicity such as flyers and media articles as well as incentives provided for participation can help send a message to the whole campus that the data are valuable for institutional improvement. We outline suggestions for increasing response rates below, as well as practices that should not be used because they can result in undue influence on participation.
Acceptable survey promotion practices
Customizing survey invitation messages with specifics for your campus.
Including small incentives and lotteries* in contact materials.
Flyers and press releases.
Unacceptable methods
More than five direct requests to participate in the survey.
Direct and personal contact by campus officials with individual students.
Revoking rights or privileges for non-response (e.g., blocking course or housing registration).
Ideas for Encouraging Student Participation in NSSE
To help you develop your own NSSE promotional materials for your campus, we have assembled examples of how colleges and universities have promoted the NSSE survey on campus. These examples include ideas for encouraging students to participate in the survey, ways schools have disseminated their NSSE results, and recruitment materials.
Suggestions for survey incentives
Offer small incentives (e.g., bookstore gift certificates or tokens for free goods or services) for each respondent. These work best if the incentive is received in advance. For example you could send a voucher by mail or by postcard that can be "cashed in" after completing the survey. These can be difficult to administer, so often institutions choose lotteries or drawing.
Enter respondents in a drawing, noting the odds of winning (e.g., the number of students invited to complete the survey and number of prizes offered) and the value of the prizes in promotional materials. Lottery prize ideas include parking permits, institution memorabilia, or tickets to athletic events.
Suggestions for general campus promotion
Post flyers on campus.
Send a press release to student and local newspaper
Submit an article, or place an ad or insert in the student newspaper.
Write an editorial for the student newspaper, by a student or administrator, explaining why the survey is important, the purpose of the survey, how the survey can be used as a tool for change on campus, and that the survey is sponsored by the institution. The editorial should emphasize that students’ participation is voluntary.
Post notices on the institutional Web site or course management systems.
Create a Facebook profile that describes the survey and ask prominent faculty or administrators to add it to their network.
Signs on campus buses or along local transportation routes.
Produce spots for campus TV and radio stations.
Set up tables in the student center or union with survey information.
For ideas of how to encourage survey participation, click here
Ethical Considerations in Survey Research
NSSE is both an institutional improvement effort and a research project, and as such, student participation in NSSE must be fully voluntary.. Participants in research studies should always be informed of their rights relative to their participation and should know any potential risks of involvement in the study. NSSE provides this information in its contacts with students (see the informed consent statement). Additional efforts you make to increase response rates should never cause students to feel they will be penalized for not participating.
The Belmont Report established guidelines that led to the creation of Institutional Review Boards (IRB) or Internal Review Boards (IRB) to regulate research involving human subjects. This report outlined three considerations for beneficence in research: (a) maximize possible benefits, (b) minimize possible harms, and (c) equitably distribute the risks and rewards of research.
Inappropriate efforts to increase participation in research fall into two general categories: coercion and undue influence. Coercive interactions are those that imply directly or indirectly that a potential participant might lose rights or privileges for not participating in the study. Explicit examples of coercive measures include requiring students to complete the survey in order to register for classes or graduate. Implicit examples include the use of language such as “you must complete this survey” or “students with real NSSEville State University pride would complete this survey.”
Undue influence occurs when the incentives used to increase participation become the primary reason why students participate, or when the number or nature of attempts to encourage participation are excessive. Students should not perceive that the compensation is so great that they
participate because they need the incentive, regardless of a sense that completing the survey would be a heavy burden to them.
Small incentives provided to each student who completes the survey are generally considered ideal. Larger incentives such as lotteries that draw prize winners from all participants may also be acceptable, but publicity for these efforts should include an estimate of the odds of winning. Survey promotion for these incentives should not emphasize the prizes to a degree that minimizes the requirement of survey participation.
References
Dillman, D. A. (2007). Mail and Internet Surveys: The Tailored Design Method (2nd ed.). Hoboken, NJ: John Wiley & Sons.
National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1979, April 18). The Belmont Report: Ethical principles and guidelines for the protection of human subject of research. Washington, DC: Department of Health, Education, and Welfare. http://www.hhs.gov/ohrp/humansubjects/guidance/belmont.html
Promotional Examples
To help you develop your own NSSE promotional materials for your campus, we have assembled examples of how colleges and universities have promoted the NSSE survey on campus. These examples include YouTube videos, flyers, posters, billboards, yard signs, painted buses, Facebook and Web postings, computer wallpaper, and more. Here are a few examples...
