Using multiple modes of survey
administration opens up the possibility of introducing
a systematic bias in the results associated with the method
of data collection. That is, do the responses of students
who use one mode (i.e., Web) differ in certain ways from
those who use an alternative mode such as paper? Further
complicating this possibility is that there are two paths
by which students can use the Web to complete the NSSE
survey: (1) students receive the paper survey in the mail
but have the option to complete it via the Web (Web- option),
or (2) students attend a Web-only school and must complete
the survey on-line (Web-only).
Using ordinary least squares (OLS) or logistic regressions
we analyzed the data from NSSE 2000 to determine if students
who completed the survey on the Web responded differently
than those who responded via a traditional paper format.
Specifically, we analyzed responses from 56,545 students
who had complete data for survey mode and all control
variables. The sample included 9,933 students from Web-exclusive
institutions and another 10,013 students who received
a paper survey, but exercised the Web-option. We controlled
for a variety of student and institutional characteristics
that may be linked to both engagement and mode. The control
variables included: class, enrollment status, housing,
sex, age, race/ethnicity, major field, 2000 Carnegie Classification,
sector, undergraduate enrollment from IPEDS, admissions
selectivity (from Barron’s, 1996), urbanicity from
IPEDS, and academic support expenses per student from
IPEDS. In addition to tests of statistical significance,
we computed effect sizes to ascertain if the magnitude
of the mode coefficients were high enough to be of practical
importance to warrant attention. Finally, we applied post-stratification
weights at the student-level for all survey items to minimize
nonresponse bias related to sex and enrollment status.
We analyzed the Web-only and Web-option results separately
against paper as shown in Table 5 by Model 1 (Web-only)
and Model 2 (Web-option) against paper. We compared Web-only
against Web-option in Model 3.
For 39 of the 67 items, the unstandardized coefficients
for Model 1 favored Web-only over paper. For Model 2,
40 of the 67 items showed statistically significant effects
favoring the Web option over paper. In contrast, there
are only 9 statistically significant coefficients that
are more favorable for paper over Web in Models 1 and
2 combined. Model 3 reveals that there are relatively
few statistically significant differences between the
two Web-based modes.
|
The effect sizes for most comparisons
in both Model 1 and Model 2 are not large -- generally
.15 or less, with a few exceptions. Interestingly, the
largest effect sizes favoring Web over paper were for
the three computer-related items: “used e-mail to
communicate with an instructor” (EMAIL), “used
an electronic medium to discuss of complete an assignment”
(ITACADEM), and self-reported gains in “using computers
and information technology” GNCMPTS).
These models take into account many student and school
characteristics. However, the results for items related
to computing and information technology might differ if
a more direct measure of computing technology at particular
campuses was available. That is, what appears to be a
mode effect might instead be due to a preponderance of
Web respondents from highly "wired" campuses
that are, in fact, exposed to a greater array of computing
and information technology.
On balance, responses of college students to NSSE 2000
Web and paper surveys show small but consistent differences
that favor the Web. These findings, especially for items
unrelated to computing and information technology, generally
dovetail with studies in single postsecondary settings
(Layne, DeCristoforo, & McGinty, 1999; Olsen, Wygant,
& Brown, 1999; Tomsic, Hendel, &Matross, 2000).
This said, it may be premature to conclude that survey
mode shapes college students' responses. First, while
the responses slightly favor Web over paper on a majority
of items, the differences are relatively small. Second,
only items related to computing and information technology
exhibited some of the largest effects favoring Web. Finally,
for specific populations of students mode may have different
effects than those observed here.
In auxiliary multivariate analyses, we found little evidence
for mode-age (net of differential experiences and expectations
attributable to year in school) or mode-sex interactions,
suggesting that mode effects are not shaped uniquely by
either of these characteristics.
Additional information about the analysis of mode effects
is available in the NSSE 2000 Norms report (Kuh, Hayek
et al., 2001) and from Carini, Hayek, Kuh, Kennedy and
Ouimet (in press). A copy of the Carini et al. paper can
is on the NSSE website. We will continue to analyze NSSE
data in future years to learn more about any possible
mode effects. |
