SAT Study

20-year Bates College study of optional SATs policy (1984-2004)
William C. Hiss ’66, Vice President for External Affairs
Prem R. Neupane ’05
A paper presented to The Trotter Group, at an annual meeting sponsored by The Nieman Foundation/Kennedy School of Government,Harvard University
Bates College, Lewiston, ME
Nov 8, 2004

Image 1

(Comments below are keyed to slide #’s on the accompanying PPT.)

Nineteen years ago, I gave with three wonderful colleagues the closing keynote session at a national conference of admissions deans and guidance counselors, a talk entitled “Admissions as Ministry.” All four of us had considered the ministry and ended up as Deans of Admissions, and our talks drew out, often with piquant humor, the parallels between the two careers. I never said better why I made Admissions my life’s work, and still remember dozens of people coming up afterwards to hug us, crying. I realized on the plane home that my suit was ruined, from all that wet make-up.

That was the year after Bates made testing optional, and I think it no accident that three of us on that panel—myself, Dick Moll at Bowdoin and Betsy DelaHunt at Sarah Lawrence—had led colleges to make testing optional. We had each thought a lot about an ethical issue: Are we helping young people with this process? Are we doing things that hurt them? We weren’t trying to point a finger at standardized testing as though it came from the forces of darkness; it hasn’t. But at least in our judgment at the time, testing was occupying too much emotional space, and kids were being hurt, either in self-esteem or in actual admissions decisions, in their access to higher ed. So we were going to try another tack, not with a statement of moral rectitude, but as an attempt to say, “How can we help kids to believe in themselves, to channel their time and energies into sluice gates of causes and values worthy of their efforts?”

It is hard to imagine, but this is the 20th anniversary of the decision at Bates to make SAT’s optional for admission, and we researched the issue for five years before recommending it to the Bates faculty, so in all, a 25-year project for this quite literal greybeard. Perhaps it is no accident that this study happened at Bates, founded by abolitionists, always co-educational, interracial and international, and fiercely independent and demanding in its intellectual life. Perhaps it is no accident that it was Benjamin Elijah Mays’ college that takes on this issue, sixty-five years after his graduation, and works on it for a generation. SAT’s were made optional in 1984; all testing was made optional in 1990. This presentation is a 20-year retrospective study of the policy, and I would praise my co-author, Prem Neupane, a senior at Bates from Nepal, who did the statistical research you will see. That students like Prem can come from very different cultures, work in a second or third language, and have a scholarly paper read at a national conference before they finish their undergraduate degree at Bates, is one mark of the success of this policy.

From the outset, Bates decided to share its data and research on the policy. We have done major research projects at roughly 5-year intervals, and provided the data and articles to any press outlets or other colleges which asked for them. Any of you who would like a packet of the research articles and clips can leave a copy of your business card with me. There has been amazing consistency of findings in our data over 20 years, and now some intriguing outcome data on our alumni, and some quite parallel and reassuring data on multicultural and international students and alumni.

The report is in 24 images on which I will have a few comments, but running like a scarlet thread through the data are three fundamental policy issues, on which people in organizations like the Trotter Group might debate as national priorities.

(1) Does requiring the tests open or truncate access to higher education? Call this a marketing issue if you like–who will apply?—or an access issue—who is allowed to go to college and where will they go?

(2) How predictive are the tests? Are they consistently predictive across populations? Are they “standardized” because people take the same test, or because their predictive value is consistent? As you will see, we seriously question the latter argument.

(3) What are the definitions of intelligence and achievement which a college (or society) signals to its youth with such a policy? What are the career and graduate degree results of our policy?

These are fundamental questions of social ethics and social policy: who gets to go to college, and what are the definitions of intelligence and achievement which a college, or a society, signals to young people by what it requires for admission? Many of us are deeply indebted to the work of two of America’s premier educational thinkers in this generation, Howard Gardner, who is across the street at the Ed School at Harvard, and the author Jonathon Kozol. Bates sees itself as being a small Petri dish of Howard Gardner’s work on the multiple definitions of intelligence, and I think Professor Gardner regards Bates as one of his small Petri dishes. I would acknowledge with profound gratitude his new gestalt on human intelligence. What you are about to see is the efforts of an “in the trenches” disciple of Gardner and Kozol.

Image 2

Some of you may know Woody Allen’s wonderful movie, “Annie Hall”, where he looks at the camera to talk about a fellow who thought he was a chicken. People tried to get his relatives to get him into some therapy to get over this obsession about being a chicken, and the relatives said, “Well, we would, but we need the eggs.”

At some levels, this discussion is about holding on obsessively, perhaps neurotically, to something that demonstrably doesn’t make any sense. We might as a gesture of health say to our young people, “You are not a chicken!”, and say to each other, “We do not need these eggs.”

It also is about being willing to take down a structure to see more clearly, to try a different way, which is the point of the Japanese haiku.

Image 3

Now we start to look at the first of our three principles, the access issue, or if you prefer, the marketing issue. The applicant pool at Bates has almost doubled, from 2200 to 4200, since we made testing optional, while admits and enrollees went up marginally.

A proposal for national debate, both at the college level and for the current “No Child Left Behind” emphasis on testing: Does testing truncate access and success more than it helps identify promise or achievement?

The most basic question for any admissions dean: Can you get a better class from twice as many applications? Of course you can, and on all the scales.

In this and previous studies, we asked statistical experts at Bates to check and critique our work. Michael Murray, a renowned international economist at Bates who designs national economies and central banking systems for third world countries, said to me, “Bill, you shouldn’t be comparing submitters and non-submitters!” I thought, “Oh no, into what statistical blind alley is Michael leading me.” He went on, “You should be comparing the enrolled non-submitters with the students you would have had to admit if you didn’t have 1500 non-submitter applicants from which to choose the very best.” He is right, of course, and at Bates and most colleges, that would comprise the entire wait list and a decent slice of the refuse pool.

Image 4

Women, international students, and U.S. multicultural students gained a lot, but all cells of our pool increased. We now have enough applications from US multicultural students to almost fill the class, and enough from abroad to fill the class twice over, and have no American citizens in the class. Many of these international applicants are some of the brightest people in their countries, and there is no question but that the influx of highly talented multicultural and international students at Bates has turned up the intellectual thermostat for the whole college. The numbers of students of color and international students are still not large, but we have increased those populations by between two and four times.

Image 5

From 1984 to 1990 about a quarter of the Bates students entered with no SATs; when all testing was made optional in 1990, the percentage of students not submitting testing rose to the mid-to-high 30% range and stayed there.

Image 6

These data are a snapshot of use of the policy by students of color, and by gender. Measurably more women than men will use the policy, and use by non-whites is about 8% higher than for non-whites. Hispanic and Black students will use the policy at a rate about 10%-15% higher than the class averages.

Something important should be pointed out here. Optional testing is often assumed to be a device for an affirmative action policy, to open the admissions process from a narrow statistical review to a more complex and subtle reading. And it does that, and as you will see, it works as a technique for affirmative action: the students succeed and graduate. But white students using the policy outnumber the students of color by about five to one. We have found that the policy appeals to all the subgroups of students which folk wisdom would tell you are the students not being much helped by standardized testing in admissions: women, rural and blue collar students, immigrants, learning disabled students, students with spike talents in something (arts, chemistry, athletics, debate, theatre, dance, political or campus leadership), and students who speak a second language, no matter what their ethnicity or citizenship. We found heavy percentages of non-submitters from Maine, because so many are rural or low-income, and have neither the money nor even the physical access to be coached for tests. But we also found an intriguing pattern of high percentages of non-submitters across the top of Maine, New Hampshire and Vermont: they turned out largely to be young people of French-Canadian heritage. They may have been US Citizens for several generations, but still speak French at home, and carry two grammars, vocabularies and syntaxes in their heads.

Image 7

Now we get to the second major point and heart of this report: What are the productive results of the policy? Over the 20-year history of the policy, the difference in Bates GPAs between submitters and non-submitters is .05 of a GPA point.

Image 8

And the difference in graduation rates is 0.1%. .05% of a GPA point, and one-tenth of one percent difference in graduation rates! On this we hang the national sluice gate system about who gets into college and where they go?

We could spend the rest of the time talking about these two slides, which are the heart of this report. In a word, in a college generally regarded as a highly demanding academic environment, non-submitters earn exactly the same grades, and graduate at exactly the same rates, as do submitters.

Image 9

At Bates, Non-Submitters enter with very marginally lower academic ratings, and marginally higher academic ratings.

Image 10

On average, Submitters score about 90 points above the Non-Submitters in Verbal SAT, and 70 points above non-submitters in Math SAT, for a total SAT gap of 160 points. This TSAT gap has been amazingly stable for the entire history of the policy, and if there reasons for that, we cannot see them.

Image 11

Yet for both submitters and non-submitters, the Admissions Office is able to read folders accurately and make very accurate predictions of success at Bates.

It is sometimes said that an optional testing policy will only work at a small college able to read applicants individually and thoughtfully. With respect, I think this is nonsense. Lots of large research universities read folders just as carefully as small colleges. Another national policy and social ethics question: What are the public costs of not admitting students who would succeed, in order to run a simple, inexpensive admissions process driven by class ranks and testing? If Bates’ experience can be extrapolated to other kinds of institutions, we may be throwing away as much as a third of our potential national talent. Many of you may know of Fairtest, the organization that has sniped away at the College Board and other testing agencies for some decades now. When Bates began this policy twenty years ago, there were a few dozen colleges on the Fairtest list not requiring or de-emphasizing testing. A few years ago I noticed the list had expanded to about 300, and now it is over 700 colleges and universities, including some major flagship publics. We badly need some rigorous research on their policies done with a common protocol, and we are in process to submit a foundation proposal for just such a joint study of optional testing colleges and universities of very different types, to see if Bates’ results hold up in a much wider study.

Image 12

While testing seems to have some very basic correlation with GPAs, non-submitters seem to outperform submitters with the same SAT scores, but for both groups, the lines are pretty flat, because virtually everyone is succeeding.

Image 13

And now to the third major point: how do wider definitions of achievement and intelligence play out in students’ choice of majors, careers and graduate fields?

There are some modest over-weights toward submitters in math and sciences, and corresponding over-weights to non-submitters in social sciences.

Image 14

The patterns of distribution by majors are intriguing, and I have made four groupings of majors in this slide. First, in three majors generally regarded as among the toughest at Bates – chemistry, biological chemistry and neuroscience – the percentages of submitters and non-submitters are very close. Second, in only three of our 32 majors is there a clear imbalance toward submitters: Math, Philosophy and Physics, but remember that when we get to the next slide. Third is a grouping of majors that folk wisdom would suggest are places that would reward imagination, intuition, unconventional thinking, interest in other unexplored cultures, new ways of viewing experience, and the like. In this group—African American Studies, American Cultural Studies, Art, Classical and Medieval Studies, Theatre, Women and Gender Studies, and Self-designed majors—there are patterns of non-submitters being equally or strongly represented. And in our largest majors—Biology, Economics, English, History, Political Science and Psychology—there are only modest trends by submitters or non-submitters.

Image 15

This slide was something of an experiment and dense with data, but intriguing. We separated by majors, and graphed GPA difference against SAT difference. In the large majority of departments, submitters and non-submitters are within .1 of a GPA point of the mean. Remember Math, Philosophy and Physics, the departments with many more submitters? Two of them have 200 point differences in SATs, but only math has a larger than average GPA difference. Physics has both a modest SAT difference and less than .1 difference in GPA.

At Bates, taking a double major is a sign of intellectual ambition. There are 313 double majors: 108 Non-Submitters and 205 Submitters, almost exactly the one-third/two-thirds ratio of non-submitters to submitters.

Image 16

Now with a 20-year time line, we can begin to look at graduate degrees and career outcomes, and there are some fascinating patterns. In general, there is very little evidence of submitters and non-submitters having different career tracks, with one glaring exception, which you will see.

In creative or human service fields like the arts, broadcasting, or education, non-submitters are represented at slightly higher rates, while the opposite is true in data processing and scientific or technical fields.

Image 17

In this slide, we divided career fields by both submitter status and by gender. Perhaps this image looks a lot better than it would have 30 years ago, but one conclusion that jumps out is that is not submitter or non-submitter that shapes more than a few career decisions, but still gender. The shapes of the graphs from top to bottom are amazingly parallel.

Image 18

Another interesting snapshot of outcomes. We isolated alumni in several specific fields, including two highly competitive fields, CEOs, (including founders, managing directors, heads of corporate divisions, etc.), and financial analysts/advisors (stockbrokers, hedge fund types, etc.). The percentages of submitters and non-submitters are about the same, and we have found the same results with business consultants, another highly competitive career. For the timid college administrators, equal percentages of submitters and non-submitters also make alumni gifts and support alumni activities. So that’s good news. But notice the splits between lawyers and doctors.

Image 19

And here is the glaring exception. Bates alumni earn graduate degrees at quite high rates: about 70% of all Bates alumni will earn at least one graduate degree. At the Master’s Degree level, the percentage of submitters and non-submitters are quite close. But in fields that require another standardized test for admission, there are big, visible gaps between submitters and non-submitters: MBAs, PhDs, MDs and JD’s.

I mean this as an honest and not a rhetorical question: are these the best, or just the best test-takers? Let that question go proxy for a lot of what we need to understand better than we do. It may be that at all three levels—K-12, college and grad school—testing truncates the number of people who will succeed given access.

Image 20

We wanted to look carefully at students of color and international students. The “n’s” are small for these groups, and I would not want to make too many sweeping conclusions, but the results seem consistently positive. You will see that where about a third of our students as a whole are non-submitters, a little less than half the African-American students and almost half the Hispanic students will use a non-submitter policy when available.

“Others” are generally bi-racial students, or in other categories that don’t neatly fit into any of the available subgroups.

Image 21

We looked also at the SAT scores and GPA’s of submitters and non-submitters in these groups. The roughly 180 point difference between submitters and non-submitters is very consistent with our students as a whole. I found it intriguing that for Black students, the non-submitters actually had a higher GPA at Bates than the submitters, if only by .16 of a GPA point. The differences in GPA’s are all with .2 of a GPA point, and with three groups have submitters a little higher, and two groups having non-submitters higher.

It is apparent that the Black and Hispanic students have somewhat lower GPA’s than the other groups, but in a college with very demanding academics, and an average college-wide GPA of less than 3.1, they are not way off the mark. And from having supervised Bates admissions for 22 years, I know from watching and working with the students that many of the Black and Hispanic students come from very different backgrounds, with multiple and substantial adjustments to make at a liberal arts college in Maine. A common pattern is that they struggle for the first year or so, and then get their feet under them, and pull their grades up in their latter two or three years: what you are seeing is their overall, four-year GPA’s.

Image 22

The proof of the pudding is graduation rate. The overall graduation rate at Bates is 87% of all entering students. All of these groups of multicultural or international students are graduating within single digits of the 87% overall mark. African-American students are graduating at about 80%, Hispanic students at about 84%, and submitters and non-submitters in all the groups are within low single digits of each other.

Image 23

Finally, we found that attendance at all graduate schools lumped together was not strongly different between submitters and non-submitters. But we found the same patterns that we saw in our population as a whole: most of the people in graduate programs that require another high-stakes test are submitters, though the numbers are so small that we cannot cite the findings as data. Grouping the various multicultural and international candidates below, MD’s, MBA’s and Ph.D’s each have 6-1 ratios in favor of submitters, while lawyers are a little less than 2-1 submitters. In contrast, for CEO’s and financial analysts, there are equal numbers of submitters and non-submitters, exactly what we found in our pool as a whole.

Let’s have a little thought experiment to expand these findings out to issues of national access to education. In California, that bell-weather state so often several years ahead of the rest of us, consider Eugene Garcia’s report of several years ago on Hispanic admission to the public institutions in California. The U-Cal public university admission rate for Hispanic students has been over the years less than 4%, and Hispanic students comprise 50% of the K-12 school cohort. Does this pass a common sense test of access to a public university system, to have a 4% admit rate for 50% of the school population? Does it pass a test of social ethics? I am not pointing a finger at California, but asking a common sense question about our country: are we getting the students the education they need to be competitive? In California, it has always been an article of faith that the state colleges and extensive community college system will provide much wider access than the state universities. But with state budget cuts caused by the “Arnold” arrangements, the curriculum of the entire community college system in California was just reduced by 4%, stranding 117,000 students seeking access to the community colleges. In Maine, we have lost startling percentages of our manufacturing jobs to overseas competition. What will societies do with people who are quite literally turned away from the door of higher education, which has been our major national route to economic improvement? In California we already see the answer: you incarcerate them. Along with several other states, California now spends more on incarceration than it does on higher education.

The same question must be asked about No Child Left Behind, which is largely driven by standardized testing results. I don’t know enough about the particulars of the testing instruments being used in No Child Left Behind to have an intelligent opinion on their reliability. But I know that Bates is one of a relatively small number of colleges that has questioned the reliability of testing for college admission, and then carefully done the follow-up research to know how predictive the tests are. Are school systems evaluating the tests they are using? Do we know if we are we truncating our success rates by using testing, and if so, how much and for whom? I would hazard a guess that at the K-12 level, we might find a result parallel to what Bates has found, that human intelligence and ambition is far more complex and multi-faceted than any standardized testing system can capture.

In his wonderful book, The Mismeasurement of Man, Stephen Jay Gould commented that all forms of standardized testing tend to reinforce existing social class lines. Our world is simultaneously less uniform and less separated from each other, than ever before. We are from very different cultures, and increasingly, living in very close quarters, either economically or physically. An old friend, something of a cynic after 30 years as a Washington insider, once commented that the reason this administration seemed so devoted to testing is that it may ensure that more of their children would be admitted at the very best colleges. I hope he is wrong, but I recall expressing frustration, while serving on a federal advisory committee during the first President Bush’s administration, that that President Bush just didn’t seem to get the issue of access to higher education. My cynic friend pointed out that President Bush had spent his entire childhood being driving to the Greenwich Country Day School in a chauffeured limo, and had never spent a single day in a public school. So what seems like a quite technical study of a quite limited subject—a single requirement for admission to college—turns out to have some real implications for who we are as a nation.