Validity is arguably the most
important property of an assessment tool. For this reason
the Design Team that developed the NSSE instrument devoted
considerable time during 1998 and 1999 making certain
the items on the survey were clearly worded, well-defined,
and had high face and content validity. Logical relationships
exist between the items in ways that are consistent with
the results of objective measures and with other research.
The responses to the survey items are approximately normally
distributed and the patterns of responses to different
clusters of items (College Activities, Educational and
Personal Growth, Opinions About Your School) discriminate
among students both within and across major fields and
institutions. For example, factor analysis (principal
components extraction with oblique rotation) is an empirical
approach to establishing construct validity (Kerlinger,
1973). We used factor analysis to identify the underlying
properties of student engagement represented by items
on The Report. These and
other analyses will be described in more detail later.
The degree to which an instrument is reliable is another
important indicator of an instrument's psychometric quality.
Reliability is the degree to which a set of items consistently
measures the same thing across respondents and institutional
settings. Another characteristic of a reliable instrument
is stability, the degree to which the students respond
in similar ways at two different points in time. One approach
to measuring stability is test-retest, wherein the same
students are asked to fill out The
Report two or more times within a reasonably short
period of time. Very few large-scale survey instruments
have test-retest information available due to the substantial
expense and effort needed to obtain such information.
It's particularly challenging and logistically problematic
for a national study of college students conducted during
the spring term to collect test-retest data because of
the amount of time available to implement the original
survey and then in the short amount of time left in the
term to locate once again and convince respondents to
complete the instrument a second time.
|
Estimating the stability aspect
of reliability is problematic in two other ways. First,
the student experience is somewhat of a moving target;
a month's time for some students can make a non-trivial
difference in how they respond to some items because of
what's transpired between the first and second administration
of the survey. Second, attempts to estimate the stability
of an instrument assume that the items have not changed
or been re-worded. To improve the validity and reliability
of The Report, minor editing
and item substitutions have been made prior to each administration.
We'll return to these points later.
Two additional pertinent indicators are estimates of skewness
and kurtosis. Skewness represents the extent to which
scores are bunched toward the upper or lower end of a
distribution, while kurtosis indicates the extent to which
a distribution of scores is relatively flat or relatively
peaked. Values ranging from approximately + 1.00 to -
1.00 on these indicators are generally regarded as evidence
of normality. For some items, out-of-range skewness values
can be expected, such as participating in a community-based
project as part of a regular course where, because of
a combination of factors (major, course selection, faculty
interest), relatively few students will respond something
other than "never."
To establish The Report's
validity and reliability we’ve conducted psychometric
analyses following all five administrations of the instrument,
beginning with the field tests in 1999. These analyses
are based on 3,226 students at 12 institutions in spring,
1999, 12,472 students at 56 institutions in fall 1999,
63,517 students at 276 institutions in spring 2000, 89,917
students at 321 institutions in spring 2001, and 118,355
students at 366 institutions in spring 2002. The following
sections describe some of the more important findings
from the various psychometric analyses of items and scales
from The College Student Report conducted between June
1999 and August 2002. Additional information about most
of the analyses reported here is available on the NSSE
web site (www.indiana.edu/~nsse)
or from NSSE project staff. |
