Non-Respondent Analysis
A frequently expressed reservation about the results from surveys is whether the people who did not respond differ in meaningful ways from respondents, especially on the questions that constitute the focus of the study. For the NSSE project, this means that non-respondents might be less engaged, for example, in some key areas such as reading or interacting with peers and faculty members, which could advantage schools with fewer respondents (i.e., they would have higher scores). As we shall see, however, this does not seem to be the case.

To determine whether respondents and non-respondents differed in their engagement in selected effective educational practices, the Indiana University Center for Survey Research (CSR) conducted telephone interviews with 553 non-respondents from 21 colleges and universities nationwide that were participating in the NSSE 2001 survey. The purpose of the study was to ask those students who had not completed either the paper or web instrument to complete an abridged version of the instrument over the phone. NSSE staff members, in cooperation with telephone survey experts from the CSR, developed two versions of the interview protocol for this purpose. Both versions contained a common core of nineengagement items. Form A of the interview protocol included six additional questions and Form B included six different additional questions. Students in the non-respondent sample were randomly assigned a priori to one of two groups. Those in Group 1 were interviewed using Form A and those in Group 2 were interviewed using Form B. This procedure allowed us to ask a substantial number of questions from the survey without making the interview too long to jeopardize reliability and validity.

CSR staff randomly selected between 100 and 200 students from each school (based on total undergraduate enrollment) who were judged to be non-respondents by mid-April 2001. That is, those classified as non-respondents had been contacted several times and invited to complete The College Student Survey but had not done so. The goal was to interview at least 25 non-respondents from each of the 21 institutions for a total of 525.

Data were collected using the University of California Computer-Assisted Survey Methods software (CASES). All interviewers had at least 20 hours of training in interviewing techniques and an additional hour of study-specific training using the NSSE Non-Respondent Interview protocol. Students with confirmed valid 		telephone numbers were called at least a dozen times, unless the respondent refused or insufficient time remained before the end of the study.

Multivariate analysis of variance was used to compare the two groups of respondents and non-respondents from the respective schools on 21 engagement and 3 demographic items from The College Student Report. The analyses were conducted separately for first-year and senior students. The total numbers of students with complete usable information for this analysis were as follows: first-year respondents = 3,470 and non-respondents = 291, and senior respondents = 3,391 and non-respondents =199.

Compared with first-year respondents, first-year non-respondents scored higher on nine comparisons. First-year respondents scored higher on only three items (using e-mail to contact an instructor, writing more papers fewer than 5 pages, and taking more classes that emphasized memorization). There were no differences on 7 of the 21 comparable items. For seniors, non-respondents again appeared to be somewhat more engaged than respondents as they scored higher on six items while senior respondents scored higher on the same three items as their first-year counterparts (using e-mail to contact an instructor, writing more papers fewer than 5 pages long, taking more classes that emphasized memorization). No differences were found on more than half (11) of the items.

Overall, it appears that undergraduate students who do not complete the NSSE survey when invited to do so may be slightly more engaged than respondents. This is counter to what many observers believe, that non-respondents have a less educationally productive experience and, as a result, do not respond to surveys. The findings from the telephone interviews suggest that the opposite may be true, that non-respondents are busier in many dimensions of their lives and don=t take time to complete surveys.

At the same time we must exercise due caution in drawing firm conclusions from these results. Telephone interviews typically are associated with a favorable mode effect, meaning that those interviewed often respond somewhat more positively to telephone surveys than when answering the same questions on a paper questionnaire (Dillman, Sangster, Tarnai & Rockwood, 1996). Thus, it appears that few meaningful differences exist between respondents and non-respondents in terms of their engagement in educationally effective practices.