Frequently Asked Questions About NSSE's Psychometric Properties

Download the PDF file

How and why was the survey developed?
NSSE was designed to assess the extent to which students are engaged in empirically
derived effective educational practices and what they gain from their college experience.
Voluminous research on college student development shows that the time and energy
students devote to educationally purposeful activities is the single best predictor of their
learning and personal development. Therefore, the main content of the NSSE instrument,
The College Student Report, represents student behaviors that are highly correlated with
many desirable learning and personal development outcomes of college.

What does the instrument cover?
The College Student Report asks students to report the frequency of their engagement in
activities that represent good educational practice. Students also record their perceptions of
the college environment associated with achievement, satisfaction, and persistence. Then,
students estimate their educational and personal growth since starting college. Finally,
students provide information about their background, including age, gender, race or
ethnicity, living situation, educational status, and major field.

Can we trust student self-reported data?
The validity and credibility of self-reports have been examined extensively. Self-reported
data is likely to be valid under five general conditions. It is: (1) when the information
requested is known to the respondents; (2) the questions are phrased clearly and
unambiguously; (3) the questions refer to recent activities; (4) the respondents think the
questions merit a serious and thoughtful response; and (5) answering the questions does not
threaten, embarrass, or violate the privacy of respondents or encourage respondents to
respond in socially desirable ways. The College Student Report was intentionally designed
to satisfy all these conditions.

Does the instrument yield valid information?
The NSSE design team that developed the instrument worked very hard to make certain
that survey items were clearly worded, well-defined, and had high face and content
validity. Logical relationships exist between the items that are consistent with the results of
objective measures and other research. The responses to survey items are approximately
normally distributed and the patterns of responses to different clusters of items discriminate
among students both within and across major fields and institutions.

Overall, the pattern of responses from first-year students and seniors suggest the items are
measuring what they are supposed to measure. For example, as one would expect, seniors
are, on average, more engaged in their educational pursuits compared with first-year
students. They also score higher on most college activities items and report that their
coursework places more emphasis on higher order intellectual skills, such as analysis and
synthesis as contrasted with memorization. Among the exceptions is that seniors reported
re-writing papers and assignments less frequently than first-year students. This may be
because first-year students are more likely to take classes that require multiple drafts of
papers or because seniors have become better writers during college and need fewer drafts
to produce acceptable written work. On the two other items, both of which are related to
interacting with peers from different backgrounds, first-year students and seniors were
comparable. Overall, the items on The Report appear to be measuring what they are
intended to measure and discriminate among students in expected ways.

Are Students’ Responses to the Survey Reliable?
Student responses to the survey are reliable to the extent that they are consistent and
reproducible. Research analysts at NSSE examined the reliability of student responses
through student-level test-retest analysis and institution-level stability analysis.

Student-level test-retest analysis. Assuming little variation in student behavior
between the test and retest, we would expect consistent or reliable responses to the
survey items. In 2002, we conducted a test-retest analysis using 1,226 respondents who
completed the same form of the paper survey twice over a several month period. For the
students’ responses on the items related to three of the benchmarks (i.e., academic
challenge, active and collaborative learning, and enriching educational experiences), the
reliability coefficients were 0.74. Student responses for the items related to student
interaction with faculty members and to supportive campus environment had reliability
coefficients of 0.75 and 0.78, respectively. These findings suggest little variation in
student responses from one testing period to the next.

Institution-level stability analysis. Assuming no major shifts in an institution’s
policies, we would expect an institution to have relatively stable or reliable benchmark
scores from one year to the next. In 2003, we conducted a stability analysis to measure
the strength of the associations between benchmark scores for 214 institutions that
participated in the 2002 and 2003 administrations of the survey. The benchmark scores
were calculated using unweighted student responses to survey items that were similar
for the two years. Values of the Spearman’s rho correlations for these benchmark scores
ranged from 0.81 (student-faculty interaction) to 0.88 (academic challenge) for firstyear
students, and from 0.83 (active and collaborative learning) to 0.93 (enriching
educational experiences) for seniors. These findings suggest that institution-level NSSE
data are relatively stable from year to year.

Do non-respondents differ from respondents?
To determine whether respondents and non-respondents differed in their engagement in
selected effective educational practices, the Indiana University Center for Survey
Research conducted telephone interviews with 553 non-respondents from 21 different
colleges and universities that participated in the NSSE 2001 survey administration.
Overall, it appears that undergraduate students who do not complete the NSSE survey
when invited to do so may actually be slightly more engaged than respondents. This is
counter to what many observers believe, that non-respondents have a less educationally
productive experience and, as a result, do not respond to surveys. The findings suggest
that the opposite may be true, that non-respondents are busier in many dimensions of
their lives and do not take time to complete surveys.

Is there mode of administration effect?
Using ordinary least squares (OLS) we analyzed NSSE 2000 data to ascertain
whether students who completed the survey on the Web responded differently than
those who responded via a traditional paper format. We controlled for a variety of
student and institutional characteristics that may be linked to both engagement and
mode. Responses to Web and paper surveys showed small, but consistent, differences
that favored the Web on a majority of items. Items related to computing and information
technology exhibited some of the largest effects favoring the Web. On the other hand,
students who answered paper surveys spent more time preparing for class and did more
reading and writing. These findings, combined with previous analysis, especially for
items unrelated to computing and information technology, are generally consistent with
the results from single institution studies.