As with all surveys, the NSSE
relies on self reports. Using self-reports from students
to assess the quality of undergraduate education is common
practice. Some outcomes of interest cannot be measured
by achievement tests, such as attitudes and values or
gains in social and practical competence. For many indicators
of educational practice, such as how students use their
time, student reports are often the only meaningful source
of data.
The validity and credibility of self-reports have been
examined extensively (Baird, 1976; Berdie, 1971; Pace,
1985; Pike, 1995; Pohlmann & Beggs, 1974; Turner &
Martin, 1984). The accuracy of self-reports can be affected
by two general problems. The most important factor (Wentland
& Smith, 1993) is the inability of respondents to
provide accurate information in response to a question.
The second factor is unwillingness on the part of respondents
to provide what they know to be truthful information (Aaker,
Kumar, & Day, 1998). In the former instance, students
simply may not have enough experience with the institution
to render a precise judgment or they may not understand
the question. The second problem represents the possibility
that students intentionally report inaccurate information
about their activities or backgrounds. Research shows
that people generally tend to respond accurately when
questions are about their past behavior with the exception
of items that explore sensitive areas or put them in an
awkward, potentially embarrassing position (Bradburn &
Sudman, 1988).
The validity of self-reported
time use has also been examined (Gershuny & Robinson,
1988). Estimates of time usage tend to be less accurate
than diary entries. However, this threat to validity can
be ameliorated somewhat by asking respondents about relatively
recent activities (preferably six months or less), providing
a frame of reference or landmark to use, such as the period
of time to be considered (Converse & Presser, 1989).
Such landmarks aid memory recall and reduce distortion
by telescoping, the tendency for respondents to remember
events as happening more recently than they actually did
(Singleton, Straits, & Straits, 1993). Requesting
multiple time estimates also makes it possible to control
for outliers, those whose combined estimates of time are
either so high that the total number of hours reported
exceeds the number available for the set of activities
or those that are unreasonably low.
Student self-reports are also subject to the halo effect,
the possibility that students may slightly inflate certain
aspects of their behavior or performance, such as grades,
the amount that they gain from attending college, and
the level of effort they put forth in certain activities.
To the extent this Ahalo effect@ exists, it appears to
be relatively constant across different types of students
and schools (Pike, 1999). This means that while the absolute
value of what students report may differ somewhat from
what they actually do, the effect is consistent across
schools and students so that the halo effect does not
appear to advantage or disadvantage one institution or
student group compared with another.
With this in mind, self-reports are likely to be valid
under five general conditions (Bradburn & Sudman,
1988; Brandt, 1958; Converse & Presser, 1989; DeNisi
& Shaw, 1977; Hansford & Hattie, 1982; Laing,
Swayer, & Noble 1989; Lowman & Williams, 1987;
Pace, 1985; Pike, 1995). They are: (1) when the information
requested is known to the respondents; (2) the questions
are phrased clearly and unambiguously; (3) the questions
refer to |
recent activities; (4) the
respondents think the questions me was intentionally designed to satisfy all these conditions.
The NSSE survey is administered during the spring academic
term. The students randomly selected to complete The
Report are first-year students and seniors who
were enrolled the previous term. Therefore, all those
who are sent the survey have had enough experience with
the institution to render an informed judgment. The questions
are about common experiences of students within the recent
past. Memory recall with regard to time usage is enhanced
by asking students about the frequency of their participation
in activities during the current school year, a reference
period of six months or less. To eliminate the variability
in week-to-week fluctuations, students report the number
of hours spent in each of six activities during a typical
week, which also allows an accuracy check on the total
number of hours students report. The format of most of
the response options is a simple rating scale, which helps
students to accurately recall and record the requested
information, thereby minimizing this as a possible source
of error.
Most of the items on The Report
have been used in other long-running, well-regarded college
student research programs, such as UCLA's Cooperative
Institutional Research Program (Astin, 1993; Sax, Astin,
Korn, & Mahoney, 1997) and Indiana University's College
Student Experiences Questionnaire Research Program (Kuh,
Vesper, Connolly, & Pace, 1997; Pace, 1984, 1990).
Responses to the Educational and Personal Growth items
have been shown to be generally consistent with other
evidence, such as results from achievement tests (Brandt,
1958; Davis & Murrell, 1990; DeNisi & Shaw, 1977;
Hansford & Hattie, 1982; Lowman & Williams, 1987;
Pike, 1995; Pace, 1985).
For example, Pike (1995) found that student reports to
gains items from the CSEQ, an instrument conceptually
similar to The College Student
Report, were highly correlated with relevant achievement
test scores (also see Anaya, 1999). He concluded that
self-reports of progress could be used as proxies for
achievement test results if there was a high correspondence
between the content of the criterion variable and proxy
indicator.
In summary, a good deal of evidence shows that students
are accurate, credible reporters of their activities and
how much they have benefited from their college experience,
provided that items are clearly worded and students have
the information required to accurately answer the questions.
In addition, students typically respond carefully and
in many cases with personal interest to the content of
such questionnaires. Because their responses are congruent
with other judgments, and because for some areas students
may be the best qualified to say in what ways they are
different now than when they started college, it is both
reasonable and appropriate that we should pay attention
to what college students say about their experiences and
what they’ve gained from them (Pace, 1984; Pascarella,
2001). |
