Episode 1: College and University Rankings

Fall means back to school and college ranking season. But does the public really know what criteria is being used to determine these "rankings" and do they really mean anything? 

Full Transcript

Steve Meredith: Hi everybody, and welcome to Solutions for Higher Education. My name is Steve Meredith and as always, I am joined by Scott L Wyatt, who is the president of Southern Utah University. 

Scott Wyatt: Hello Steve, it's nice to be with you this beautiful day. 

Meredith: Welcome, and as you mentioned, it's a spectacular autumn day in Cedar City, Utah, which by the way, for those of you listening, if you've never been to Cedar City, there's no prettier place in the United States, I think I can say without fear of contradiction. 

Wyatt: It is a gorgeous place. 

Meredith: Fall means back to school, which is important to you and I since we work in that business, and it's also the college ranking season for football and everything else, including U.S. News and Kiplinger and the Princeton Review. They rank colleges and universities, and I know you have pretty strong feelings about that. 

Wyatt: Yeah, so you look at all these groups that do rankings and then they publish it, and most of the members of the public—potential students or students, family members—they see these rankings, but they don't know they criteria that's being used. 

Meredith: So how did SUU do in the rankings?

Wyatt: I guess it depends on how you look at it. The strict question of how did Southern Utah University do on the rankings? We did great. If we were to look at all of the public regional universities that are included within the eight intermountain west states, SUU came in third out of 8—that's pretty good—and if you focus it down just to the Utah schools, SUU is the highest ranked public regional university in Utah, so that's all good news.

Meredith: Sure.

Wyatt: But if you look deep into these rankings, it would cause anyone to be troubled. 

Meredith: How so? 

Wyatt: What are the criteria that are being used to evaluate us? And if you look carefully, what you see is U.S. News is ranking the universities based on things like reputation, faculty resources, student selectivity, alumni giving, endowment size. Have you heard anything yet that suggests whether students are learning? 

Meredith: No, it sounds like it's a measurement of how elite the institution is, or how many resources it has. 

Wyatt: Yeah, it's about potential, but it's not about actual. And so we are judging all of these universities and publishing these rankings based on our potential to do good perhaps, but not on whether we are doing any good, whether we are changing any student's lives, whether we are helping a student learn, became a more creative thinker, to be more prepared for the democracy that she or he is going to be a part of, whether the students can get a job. None of these things are in the rankings. So why is it that we care so much about student selectivity? You know, that's really great - - "you've turned a lot of students away, so you must be terrific." Is anybody actually learning?

Meredith: It seems like to me that the best measure of learning would be to take an average student and have them turn out to be outstanding rather than to take an already outstanding student and graduate them, right? I mean, doesn't that seem to make sense? 

Wyatt: Right, so you know, you bring somebody into your gym who was already a model of physical fitness, spend a few months with them and send them out looking the way they looked when they came in, and everybody is impressed with how good your gym is. But bring in somebody who looks a little less fit and help them improve and measure the distance, measure the delta, measure how much improvement has taken place, and then you know you've got a good trainer at this gym. What we're doing at universities largely is we're actually not measuring fitness when they leave, and we're not measuring the improvement of fitness while their there, so it's a troubling business in my mind. 

Meredith: So, smaller universities like SUU participate in these rankings. Do we not have any leverage or any sway on what criteria are used? 

Wyatt: Yeah, I think that most people would say 'Hey, you know what? It's Princeton Review or it's U.S. News or it's whoever else and we don't have any control over that,' but the reality is that we have enormous control over that if we choose. We're filling out surveys and sending them back, we're giving them data from which they judge us, and we're culpable. We're part of this whole thing, and I suspect the same schools that do very well are thrilled about it, and continue playing the game and the rest of the schools feel obligated to stay in, too. But if we as a community said, "You know what? We're not interested anymore about being evaluated on an elitist sort of a standard based on how much money you've got." I don't want to suggest that Southern Utah University doesn't have a lot of resources, because we do. We actually ranked very high on U.S. News and World Report rankings, but that's not what we care about. What we care about is that we're actually changing lives. That we're actually helping people grow, develop. Success should be measured by whether we're adding value to some person or some family—that we're actually doing good. And nobody is asking that question. Nobody amongst the ranking organizations, they're not really asking that question. 

Another challenge, Steve, with these rankings, is that there are a lot of apple to orange comparisons that don't hold fair. So, for example, private schools tend to be higher ranked, and part of the reason that they do is because one of the criteria in most rankings is the size of the endowment or contributions to an endowment. But this is an interesting piece, because they don't rank state contributions. So, for Southern Utah University, the State Legislature funds us every year the amount of money that a private school would get from an $875 million endowment. The private school has to have an $875 million endowment to generate revenue without touching the principle every year equal to what the Utah State Legislature gives us freely. But the private school gets benefit in the rankings for that endowment, and the public school gets no acknowledgement of this tremendous public support, which is as important, or more important than the endowment. And there are some states where the legislature is contributing 8% of the higher education budget, or 20%, 25%, or 12%, whatever the number is, but I think that's…that reveals one of these little challenges that causes people to assume that private schools are so much better because they come out so much higher in the rankings, and one of the reasons they do is because of this financial contribution.

Meredith: So, what criteria, then, would you suggest that they use? 

Wyatt: Well let's walk through this—that's a great question, Steve. What they're ranking us on is perhaps what might be thought of as the simplest or the most 'status quo' sort of method, which is, "You have a lot of money, so you must be great." But, if we were to take that one more step, the questions would be, "Well how are you spending your money? Are you spending your money on practices that are proven to lead to higher educational outcomes?" That would be at least a step in the right direction. The American Association of Colleges and Universities (AAC&U) has done tremendous research on this, and they have a list of high impact practices that include things like:

• Learning communities;
• writing intensive courses;
• collaborative assignments;
• undergraduate research;
• internships;
• service learning;
• capstone courses;
• and a whole list of these things that are proven to have better outcomes.

So, if we just shifted a step to the side and said, "We're not going to ask you how much money you've got, or if we do, we're going to at least pursue the next step which is, 'How are you spending the money?'" And it's not whether there are internships available because every university has that, but "What percentage of your students actually get the benefit of an internship?" And, "What is the internship like?" And, "Is there an opportunity for the students to reflect on what they've learned and really internalized something?" You know, experience really isn't a great teacher. Reflecting on the experiences we have are (or is) a great teacher. So that would be one step better.

But Steve, we could do even better than that. We're the masters of assessment, I mean, that's what we do. Nobody should be able to do this better than us. If we shifted another step toward the goal that would take us right into actually assessing whether the students are learning. Now, we would ask those questions. The first semester the freshman show up, we would be asking them to take a writing test to see how good they are. We would ask them to take a quantitative literacy class. We would ask them to show us something about reading comprehension. We would just do all of these kinds of things, and then on graduation, we would ask them all the same questions. We would test the delta, we would see how much change there's been. That's what we should be doing. 

Meredith: So, it's not just resource availability, but also resource allocation and use and what the outcomes from those resources are. Is that what you're saying? 

Wyatt: Right. It's the resources we have, how we're employing them, and what the outcomes are. But sometimes I even wonder if we should be asking the question about resources, which is the only thing, or the primary thing that these groups like U.S. News are asking. Because it's America after all. We believe in Cinderella stories, the underdog. We believe that people can grow up in disadvantaged situations and succeed. And so why don't we apply that to universities and colleges and say, "You know, it's possible that a university with a small endowment can actually do a better job than a university with massive endowment, because there's something about them. They've got grit. They've got tradition. They've got faculty that really care and are focused on learning outcomes, that are exploring this. They're being creative".

I think that assessing this at all based on the amount of money we have is completely flawed. We should be looking at improvement, value added, how the students come out. If every student that comes has a 36 ACT score and they all leave brilliant, then maybe they've learned, maybe they already were smart and would have picked it up anyway. 

Meredith: It seems to me that the proposal you're making here would result in rankings that would be much more useful to prospective students and their parents, as they consider what ultimately is one of the largest expenditures that they'll make during their lifetime. 

Wyatt: Right. It's, "What's my return on investment? Are you going to help me become a better critical thinker or not?" That's the question.