Thursday, November 15, 2001

New Data Released on Student 'Engagement,' but Campus-Specific Figures Are Held Back By THOMAS BARTLETT Jodi H. Levine, assistant vice provost for university studies at Temple University, sums up the views of many college administrators when she says the National Survey of Student Engagement "speaks the language of what college is about." But if that is so, few can hear what it is saying. That's because college officials won't let data about individual institutions be released. They may praise the survey -- known as "Nessie" after its acronym, NSSE -- but they want only average national data released. Like its mythical namesake, they believe it should be kept out of public view. Hence, the names of colleges are not listed with their individual scores. Parents and students looking for a copy would be out of luck unless they went searching for data on a public college -- and even then they would succeed only if they filed a laborious freedom-of-information request. So much for a report that was started just last year as an alternative to the much-maligned U.S. News & World Report rankings. Chalk it up to academe's fear of accountability, says Alvin P. Sanoff, a higher-education consultant and former managing editor of the U.S. News guide. "Colleges love to sit there and take potshots at U.S. News, but when it comes to alternatives, there is nobody home," he says. The results of the National Survey of Student Engagement were mailed to participating colleges this month. The $3.3-million project, cosponsored by the Pew Forum on Undergraduate Learning and the Carnegie Foundation for the Advancement of Teaching, gauges how colleges encourage learning by scoring the responses of freshmen and seniors to 41 questions. Now in its second year, the survey has collected responses from 155,000 undergraduates at 470 four-year institutions over two years. It is an all-consuming task for its director, George D. Kuh, a higher-education professor at Indiana University at Bloomington. But it is also one that leaves Mr. Kuh in an awkward position. Frequently the parent of a prospective college student will call the survey's headquarters and ask for a copy of the report. A staff member then has to explain that, because of a confidentiality agreement, the scores can only be released by the colleges themselves. "The parents say, 'Well, what use is the survey then?'" Mr. Kuh says. It's a valid question, and one Mr. Kuh hears all too often. But here's his dilemma: Most colleges would be unwilling to participate if they knew the results would be made public. On the other hand, it is difficult, as Mr. Kuh says, to "change the conversation" in higher education when the data that are supposed to make that change never see the light of day. And the reluctance from colleges is strong. This was writ large by an e-mail message from a college official that circulated among administrators at several colleges after the 2000 survey was completed. The e-mail message criticized Mr. Kuh's decision to publish a list of just four liberal-arts colleges -- colleges that had scored well in all five of the survey's benchmarks. The message, forwarded to Mr. Kuh by one of his former students, said he had "gone over to the dark side" and was trying to "manipulate the media," according to Mr. Kuh. Mr. Kuh's sin was naming names. Never mind that all the information was positive and never mind that each of the four colleges had given permission for its name to be used. This year's survey does not list the top performers or any of the individual college scores. Along with institutional data meant only for the eyes of officials from those institutions, the survey lists anonymous, national statistics, although some data are analyzed by type of institution. Unlike the U.S. News rankings, the student-engagement survey was not intended to provide year-to-year comparisons, although NSSE officials say that in the future this may change. Since this year's results were combined with last year's, some of the findings are not surprising. Many of the averages are similar to last year's totals, which, according to Mr. Kuh, indicates that the survey's methodology is sound. Among them: Nearly half of seniors have never written a paper that is 20 pages or longer. When asked if their institution provided the support necessary to succeed academically, about one out of four freshmen answered "some" or "very little." The statistic most touted by Mr. Kuh shows that students are spending less time studying than previously thought. About 20 percent of freshmen and seniors spend 5 hours or less per week, while only about 15 percent spend 16 or more hours hitting the books. The survey also shows that students find smaller colleges more engaging. Institutions with fewer than 4,000 students scored better than larger colleges in every category, from the level of campus support to the degree of academic challenge. While the survey thus far has offered little in the way of public accountability, it has provided a useful "reality check" for many colleges. The Georgia Institute of Technology, for example, after receiving the 2000 survey results, created a $250,000 grant fund to encourage students and faculty members to participate in joint research projects. But not every college has found the survey useful. Colgate University is a telling example. Michael A. Cappeto, dean of the college, calls the survey a "wonderful, well-crafted tool." Nonetheless, Colgate, which participated in the 2000 survey, chose not to take part in 2001 and does not plan to participate in the future. "We didn't find the information to be of value to us." The reason, according to Mr. Cappeto, is that "none of our peer schools participated." Only a handful of the colleges considered "top tier" by U.S. News have taken part in the survey. Asked why so few elite colleges have participated, Mr. Kuh becomes coy. "I could think of some reasons," he says, without elaborating. Mr. Sanoff is more frank. "It's what I call arrogance," he says. "I think it reflects a feeling that, hey, we're not having any trouble filling our classes and meeting our financial targets, so why make the extra effort, even if that effort might shed some light on our strengths and weaknesses and help us improve." Unlike Colgate, American University found the results both gratifying and helpful. While Karen F. Jones, director of institutional research, condemns the U.S. News guide as "basically a popularity contest," she says the student-engagement survey is "very helpful." "It was a nice confirmation for us that we were doing what we wanted to do," she says, while declining to go into specifics. But when asked about making the information public, her tone changes. "I don't think I would be comfortable with that kind of scenario," she says. Ms. Jones says releasing the results might place too much emphasis on the survey and discourage prospective students from, among other things, making campus visits. She declined to elaborate. "I think it's better to have this as an internal document," she says. Mr. Sanoff argues that there is such reluctance because colleges view the data as a "trade secret." "Schools have a lot more information that is of interest to students and families than they're willing to provide," he says. "They have a way of making matters of commerce sound like matters of high principle." While American University's results were a "nice confirmation," Southwest Texas State University's report was not as encouraging, according to Robert Smallwood, the associate vice president for academic affairs at the college in San Marcos. "We are not a model of excellence," Mr. Smallwood says. "We are really disappointed and frustrated by our outcomes. But we think NSSE is a tool that can help us get where we want to be." Not surprisingly, Mr. Smallwood declined to elaborate. And he is one of the survey's most enthusiastic proponents. Southwest Texas has participated in the survey both years it has been conducted and plans to participate in coming years. "We're really excited about the dialogue this has stimulated within our college" on making students more engaged, he says. Like Ms. Jones at American, Mr. Smallwood would prefer not to have the data made public. The difference is that, because Southwest Texas is a state institution, its results could likely be obtained under the state's sunshine laws. Though it seems doubtful that a prospective student would go to that much trouble. "I think it's best used as a diagnostic tool within our college," Mr. Smallwood says. It's an assertion that Milton Greenberg, a professor emeritus of government at American University and former provost, has heard before. Says Mr. Greenberg: "It's the same excuse we use for everything: 'Oh, it's for self-review.' I think it's a dodge." Not every college is shy about the survey. A few, like Bowling Green State University, in Ohio, are more forthcoming. Bowling Green posted the results of its 2000 survey on the university's Web site. Officials plan to post the 2001 results as well. This seems natural to William E. Knight, the university's director of planning and institutional research. "Our default position is that if there's any question, go ahead and make it public," he says. For instance, on the positive side, 85 percent of undergraduates in the survey rated their overall educational experience as good or excellent. But the survey also found that 80 percent of students never or only occasionally discussed ideas from their readings with faculty members outside of class. Mr. Knight is among those who have reservations about the U.S. News rankings. But he thinks academics are "speaking out of both sides of our mouths. ... Shame on us if we have a measurement tool that does meet our values and we don't share it publicly." Some supporters of the survey advocate incorporating the results into the U.S. News rankings. So far, no progress has been made. "What they're doing is a really valuable effort, but they can't provide us with any data," says Anne McGrath, special projects editor for education at U.S. News. Mr. Kuh counters that the magazine could pressure colleges by requesting student-engagement data, something U.S. News has not done thus far. At a recent dinner celebrating the release of the second report, Mr. Kuh and others involved with the survey discussed how colleges are using the information to make institutional improvements as well as the results of the nationwide findings. But they also made no secret of their frustrated desire to get this information out of filing cabinets and into the hands of prospective students. "We hope that institutions will be enlightened enough to realize that this is an age of disclosure," says Peter Ewell, a member of the survey's national advisory board. If the reaction of many college leaders is an indication, the age of enlightened disclosure isn't coming anytime soon. Says Ms. Jones, of American: "I know there's a lot of demand to release these results, but I just don't think it's going to happen." Background article from The Chronicle: Are Students Actually Learning? (11/17/2000)

Easy-to-print version

E-mail this story