BY BRIAN C. MITCHELL

Last month’s Huffington Post College article (also published on Academe), “Grading Higher Education: When Worlds Collide,” sparked a lively offline discussion and is worthy of additional exploration.

In that post, we examined how various magazines, journals, and other outlets grade American higher education in their version of what some call the higher education “swimsuit edition.” We noted that various perspectives often shape the approach utilized.

We argued that many ranking strategies are heavily based on inputs causing the numerical rankings to change slowly from year to year because they rely more heavily on reputation and selectivity.

Based on a study by James B. Stewart in the New York Times in October examining the evolution of the higher education literature on this topic, we concluded that Mr. Stewart demonstrated convincingly that measurable outcomes — more attuned to the metrics sought by most American families – have lasting value.

With the growth in popularity of outcomes-based rankings assessments, we offered the following thoughts:

• American higher education is moving closer to center stage visibility in the court of public opinion
• The decades-old disagreements between higher education officials and the editors of US News over issues like methodology is at best “navel gazing,” and
• The new battleground is likely to be over outcomes-based surveys.

The most interesting response came from a highly respected, distinguished colleague, Dr. Alexander Astin, the Allan M. Carter Professor Emeritus and Founding Director of the Higher Education Research Institute at UCLA. Many of us use Dr. Astin’s survey research in our work on student life in our own institutional planning as the basis for subsequent research.

Dr. Astin finds: “the pecking order of American higher education institutions that drives the annual ‘admissions madness’ in this country has little to do with rankings and ratings. It is, rather, part of our culture, part of our shared beliefs about which are the most ‘excellent’ institutions. These beliefs have remained largely unchanged for more than 50 years. What US News and others have been trying to do is simply to codify – put numbers on – these shared beliefs.”

Dr. Astin also notes that while there is considerable value to the “outputs” approach that is gaining ground, the cold fact is that no one has the data required to properly look at outputs. He uses research on earnings as an example.

Dr. Astin reports that decades of research on earnings suggest that it is not simply dependent on the level of a student’s ability when admitted as a freshman but on other factors, including career choice, major, parental occupation, degree aspirations, and social class, among others. Each carries its own set of biases.

Accounting for such bias, a researcher would need to calculate an “expected earnings” for each entering student to measure against their actual earnings. Astin suggests: “By aggregating these expected earnings and actual earnings figures for all students entering a particular college, you’re able to obtain a much more valid estimate of that college’s effect.” He further argues that earnings is only one potential output and must be grouped with other outcomes to assess the value of a college education.

Whatever the approach, Dr. Astin regrets the emphasis placed by federal and state officials as well as US News-type rankings on degree completion rates.

He reports that degree completion rates are largely dependent on the level of the academic preparation of entering freshmen. As such, they are an indirect measure of SAT/ACT scores making them in turn a reflection of shared cultural beliefs about the pecking order.

Put in other terms, students with high test scores prefer elite colleges because they – and their families – believe that these colleges are “the best.”

Dr. Astin’s comments are a cautionary tale for those who are interested in seeking broad, standardized measures of excellence and value. His words add a cultural, social, and psychological dimension to a “paint by numbers” approach. He does not reject the value of a blended approach to combine inputs and outputs in measuring quality. But he is correct to argue for a longitudinal approach that adds nuance and perspective to data on the value of higher education.

One fact is clear. The scattershot and prescriptive efforts now utilized are insufficient and a poor basis on which to develop sound public higher education policy.