We used information from the
focus groups and psychometric analyses to guide revisions
to the 2001 version of The College Student Report. We
also worked closely with survey expert, Don Dillman to
redesign the instrument so that it would have a more inviting
look and feel. For example, we revamped the look by substituting
check boxes for the traditional bubbles so the instrument
looked less test-like. These and other changes created
a more inviting feel to the instrument. We then did cognitive
testing on the instrument via interviews with Indiana
University undergraduates in mid-November 2000 as a final
check before beginning the 2001 survey cycle.
The group, 14 men and 14 women, was recruited by the Center
for Survey Research (CSR) staff. CSR and NSSE staff members
worked together to draft the interview protocol, study
information sheet, and incentive forms, all of which were
approved by the Indiana University Bloomington Institutional
Review Board, Human Subjects Committee. Students were
compensated $10 for their participation. CSR professional
staff and NSSE associates conducted the interviews. Interviews
lasted between 30 and 45 minutes and were tape recorded
with respondent permission. The interviews were subsequently
transcribed and
analyzed by two NSSE staff members. Included among the
key findings are:
|
- The vast majority of students indicated that the
instrument was attractively formatted, straightforward,
and easy to read, follow, and understand. Most agreed
that they would probably complete the survey if they
were invited to do so, though four students said that
the survey length might give them pause.
- All of the respondents found the directions and
examples helpful.
- The majority of students interpreted the questions
in identical or nearly identical ways (e.g., the meaning
of primary major and secondary major, length of typical
week).
- Several students were not entirely sure who was
included in the survey item dealing with relationships
with administrative personnel.
- Of the 20 students who discussed the web versus
paper survey option, nine indicated that they would
prefer to complete the survey via the web. Reasons
for preferring the web included that it was "more
user-friendly... more convenient... easier." However,
nine other students indicated that they preferred
the paper version, and the remaining two students
were undecided. This suggests that it is important
to offer students alternative modes to complete the
survey.
Summary. The results of
the cognitive interviews suggest that respondents to The
College Student Survey understand what is being asked,
find the directions to be clear, interpret the questions
in the same way, and tend to formulate answers to questions
in a similar manner. NSSE staff used these and other results
from the cognitive testing to make final revisions to
the instrument for 2001. These revisions included several
minor changes that were mostly related to formatting of
response options and a few wording changes. |
