Frequently Asked Questions About NSSE's Psychometric Properties

Download the PDF file

How and why was the survey developed?
NSSE was specifically designed to assess the extent to which students are engaged in empirically derived good educational practices and what they gain from their college experience. Voluminous research on college student development shows that the time and energy students devote to educationally purposeful activities is the single best predictor of their learning and personal development. Therefore, the main content of the NSSE instrument, The College Student Report, represents student behaviors that are highly correlated with many desirable learning and personal development outcomes of college.

What does the instrument cover?
The College Student Report asks students to report the frequency with which they engage in activities that represent good educational practice. Students also record their perceptions of the college environment associated with achievement, satisfaction, and persistence. Then, students estimate their educational and personal growth since starting college. Finally, students provide information about their background, including age, gender, race or ethnicity, living situation, educational status, and major field.

Can we trust student self-reported data?
The validity and credibility of self-reports have been examined extensively. Self-reports are likely to be valid under five general conditions. They are: (1) when the information requested is known to the respondents; (2) the questions are phrased clearly and unambiguously; (3) the questions refer to recent activities; (4) the respondents think the questions merit a serious and thoughtful response; and (5) answering the questions does not threaten, embarrass, or violate the privacy of the respondent or encourage the respondent to respond in socially desirable ways. The College Student Report was intentionally designed to satisfy all these conditions.

Does the instrument yield valid information?
The NSSE design team that developed the instrument worked very hard to make certain the items on the survey were clearly worded, well-defined, and had high face and content validity. Logical relationships exist between the items in ways that are consistent with the results of objective measures and with other research. The responses to the survey items are approximately normally distributed and the patterns of responses to different clusters of items discriminate among students both within and across major fields and institutions.

Is the instrument reliable?
Overall, the pattern of responses from first-year students and seniors suggest the items are measuring what they are supposed to measure. For example, as one would expect, seniors are -- on average -- more engaged in their educational pursuits compared with first-year students. They also score higher on most college activities items and reporting that their coursework places more emphasis on higher order intellectual skills, such as analysis and synthesis as contrasted with memorization. Among the exceptions is that seniors reported re-writing papers and assignments less frequently than first-year students. This may be because first-year students are more likely to take classes that require multiple drafts of papers or because seniors have become better writers during college and need fewer drafts to produce acceptable written work. On the two other items, both of which are related to interacting with peers from different backgrounds, first-year students and seniors were comparable. Overall, the items on The Report appear to be measuring what they are intended to measure and discriminate among students in expected ways.

Is the instrument stable?
Reliable measurement tools provide consistent readings from one administration to the next. In an effort to determine stability, we’ve used three approaches to establish if students at the same institutions report their experiences in similar ways from one year to the next.

The first stability estimate, a correlation of concordance, was based on student responses at common institutions that participated in NSSE at various time periods (N=127 for 2000 and 2001; N=156 for 2001 and 2002, N=144 for 2000 and 2002). We computed Spearman's rho correlations for the five benchmarks using aggregated institutional level data. The benchmarks and their rho values range from .74 to .92 for the 2000-2001 comparison, .79 to .92 for the 2001-2002 comparison, and .76 to .90 for the 2000 and 2002 comparison. These findings suggest that NSSE data at the institutional level are relatively stable from year to year.

A second analysis of stability, using matched sample t-tests to determine if differences existed in student responses, was also conducted. For both first-year and senior students, only 3% of NSSE items between 2001 and 2002 have large effect size differences. Similarly, about 18% of the items between 2000 and 2001 have large effect size differences and less than 16% of items common to 2000 and 2002 have large effect size differences. For both first-year students and seniors, NSSE items are highly or moderately correlated with coefficients ranging from .60 to .96.

The third approach to estimating stability was test-retest analysis. Using Pearson product moment correlation, the overall test-retest reliability coefficient across all items for all students (N=569) who completed The Report in 2000 was a respectable .83. A second study of the test re-test reliability examined coefficients for all students (N=1,226) across five engagement scales. The coefficients ranges were also quite stable from .74 to .78.

Taken together these analyses indicate that the NSSE survey is quite stable from one year to the next.

Do non-respondents differ from respondents?
To determine whether respondents and non-respondents differed in their engagement in selected effective educational practices the Indiana University Center for Survey Research conducted telephone interviews with 553 non-respondents from 21 different colleges and universities. Overall, it appears that undergraduate students who do not complete the NSSE survey when invited to do so may actually be slightly more engaged than respondents. This is counter to what many observers believe, that non-respondents have a less educationally productive experience and, as a result, do not respond to surveys. The findings suggest that the opposite may be true, that non-respondents are busier in many dimensions of their lives and don't take time to complete surveys.

Is there mode of administration effect?
Using ordinary least squares (OLS) and logistic regressions we analyzed NSSE 2000 data to ascertain whether students who completed the survey on the Web responded differently than those who responded via a traditional paper format. We controlled for a variety of student and institutional characteristics that may be linked to both engagement and mode. These variables were: class, enrollment status, whether living on campus, sex, age, race/ethnicity, major field, Carnegie Classification, public or private, undergraduate enrollment, admissions selectivity, urbanicity, and academic support expenses per student. Responses of college students to Web and paper surveys showed small but consistent differences that favored the Web on a majority of items. Items related to computing and information technology exhibited some of the largest effects favoring Web. These findings, especially for items unrelated to computing and information technology, generally dovetail with single institution studies.