(Author's Note: This post originally only included the top 50, but has now been expanded to include the entire top 100)Leaked recently are copies of the 2010 US News law school rankings, providing yet another example of why journalists should not be charged with composing an index. US News collects data on quality assessment (survey data), selectivity, placement success, and faculty resources, standardizes each measure and weights it appropriately to create a final score ranging from 1 to 100. Obviously, any ranking will have it's shortcomings, but US News has done a spectacular job of creating a ranking which is nothing short of total and complete nonsense. Before diving into criticism, a disclaimer: I am not a law student nor am I a journalist, I am a political science PhD student who deals with issues of measurement and quantification regularly. Additionally, I have a master's degree in political science with a minor field in methodology. The ranking replacement I provide at the end of this post does not feign to be in any way adequate, but is far more accurate a portrayal of these institution's quality than that provided by the magazine.
- Quality Assessment - This may be the most potentially relevant measure US News includes, as what matters more than the perception of your potential future colleagues and employers? Some schools will claim particular biases within the data against certain regions (an east-coast bias or a bias against the south-east). Given that US News claims to be a national ranking, those so-called "biases" are not relevant. Unfortunately, US News does not tell us how the survey was scored, what weights were used, clustering, etc. It is not unlikely that they simply took the raw average result without applying any relevant weights, leading to potentially biased results (particularly given the 31% response rate among professionals). Furthermore, US News does not tell us the number of respondents nor the variation in responses, all necessary pieces of information to determine the accuracy of their scores.
- Selectivity - First, "GPA" -or- "how to use a measure that is completely irrelevant". The US News data is measured by median undergrad GPA, comprising 12.5% of the entire ranking. However, no controls exist in measuring undergrad GPA across different institutions. US News is assuming that a 3.9 from the University of Mississippi is equivalent to a 3.9 out of Yale. Please, you're kidding right? Second, they include acceptance rates. Unfortunately, these are also difficult to compare across schools given differences in class sizes. Smaller schools will inherently have lower acceptance rates without regard to whether those students are provided a better education. This measure would be relevant only if the rankings controlled for institutional size.
- Placement Success - Again, also not relevant. All jobs are not created equal, and a job with a firm in Beaumont, Texas is slightly different from one in Lower Manhattan. If this glaring oversight were not enough the reported employment rates are not specific to law jobs. A school may count their student employed whether they are working for a large firm, a judge, Starbucks, or the local meat packing plant. I would hope students are actually interested in getting law jobs after 3 years of school and a truck load of debt. Second, the most irrelevant of all measures: Bar Passage Rate. Like GPA, this is simply not comparable. You cannot equate passing the Wisconsin Bar to passing the California Bar, the Wyoming Bar as identical to the New York Bar, or the Louisiana Bar as no different from the Virginia Bar. If passage rate is to be included it must be weighted by the state bar being taken.
- Faculty Resources - A potentially helpful measure, but most of these statistics US News does not release in the issue. It includes financial aid, expenditures per student on instruction, library services, library volumes, etc.
What's left is the 53% of the initial US News ranking that is provided and does not exist in the realm of total absurdity. The ranking is listed below. Notably, the top 15 remains relatively stable (no one enters or leaves, though UCLA does step down from 15 to 16). The further you get from number 1, however, the more innacurate US News becomes. Within the top 50, three schools are over or under ranked by 10 or more. UC-Hastings should be 10 ranks above where it currently is, but that is the price they pay for boycotting US News one year. Indiana is overranked by 10, but I'm sure the US News' claims that they are tied with Illinois will come as a shock to most Midwesterners.
Finally, the greatest disparity of any school between the below rankings and those of US News top 50 is Alabama. Alabama provides a valuable example of why the US News rankings are so flawed. Based on any sensible measure, Alabama should be ranked in the 40's. So why the sudden jump to 30? Why is number thirty having such a hard time getting their students law related jobs? Looking at the data, Alabama achieves this incredible accomplishment by gaming the absurdities of the US News system (which everyone knows they've been doing for some time in attempts to improve their reputation, which is number 43 in the survey). To decrease their acceptance rate, they've begun offering prizes to students who apply, such as IPods. Clearly this will increase your application rate and correspondingly decrease your acceptance rate, while maintaining absolutely no relevance to the quality of your institution. Secondly, Alabama does not attract students from as equally difficult undergraduate institutions as many of the competing schools throughout the thirties and forties, unless you actually believe that an undergrad at Alabama is equal to an undergrad at top tier Georgia or North Carolina. Dream On. To make matters worse, Alabama Law has announced that it will accept students from the University of Alabama Honors College without requiring an LSAT. Why would they do such a thing? Obviously, despite their high GPA's, a student at Alabama Honors College isn't going to do well on the LSAT because, frankly, they're not that smart. If they were, they would have gone to Auburn. Furthermore, various rumors have circulated regarding Alabama's improperly reporting student's final GPA's upon completion of undergraduate.
The absurd over-ranking of Alabama should come as no surprise, they are experts at gaming the system no matter the area (see BCS 2008). However, eventually you get caught and embarrassed, just like losing to Utah in the Sugar Bowl. The comparison between the law school rankings and the BCS is a good one, as both rankings are completely absurd and a failed measure of actual quality.
Biggest Gainers:
1. (+21) - University of Oregon
2. (+17) - Catholic University
3. (+13) - Temple University and University of Pittsburgh
Biggest Losers:
1. (-19) - Louisiana State University and University of Nevada - Las Vegas
2. (-17) - Pepperdine University
3. (-16) - University of Buffalo - SUNY
The 53% Ranking (Difference Between Ranking and US News)
1. Yale (0)
2. Harvard (0)
3. Stanford (0)
3. Columbia (+1)
5. Chicago (+1)
6. New York University (-1)
7. University of Virginia (+3)
8. University of California - Berkeley (-2)
8. University of Michigan (+1)
10. University of Pennsylvania (-1)
11. Duke University (-1)
11. Georgetown University (+3)
13. Cornell University (0)
14. Northwestern University (-4)
15. University of Texas (0)
16. University of California - Los Angeles (-1)
16. Vanderbilt University (+1)
18. Washington University - St. Louis (+1)
18. University of Southern California (0)
18. Emory University (+2)
21. George Washington University (+7)
22. University of Minnesota (-2)
23. Boston University (-3)
23. University of North Carolina (+7)
23. University of Notre Dame (0)
26. Boston College (0)
27. Washington and Lee (+3)
27. University of Illinois (-4)
29. University of California - Hastings (+10)
29. University of Iowa (-3)
29. University of Wisconsin (+6)
32. College of William and Mary (-4)
33. Ohio State University (+2)
33. University of Indiana (-10)
35. University of California - Davis (0)
35. Wake Forest University (+5)
35. Fordham University (-5)
38. University of Washington (-8)
39. University of Arizona (+4)
39. Tulane University (+6)
41. University of Georgia (-6)
42. University of Colorado (+3)
43. University of Alabama (-13)
43. Brigham Young University (-2)
43. American University (+2)
46. University of Maryland (-3)
47. University of Florida (+4)
47. George Mason University (-6)
49. Arizona State University (+6)
50. University of Utah (-5)
51. University of Connecticut (+1)
52. Case Western Reserve University (+3)
52. Temple University (+13)
52. Southern Methodist University (-3)
52. Yeshiva University (-3)
56. Villanova University (+5)
56. University of Oregon (+21)
58. University of San Diego (+3)
58. University of Pittsburgh (+13)
60. University of Kansas (+5)
61. University of Miami (+10)
61. University of Houston (-2)
61. Florida State University (-9)
61. Lewis and Clark College (0)
61. University of Missouri (+4)
61. University of Cincinnati (-9)
67. Brooklyn Law School (-6)
67. University of Kentucky (-12)
67. Baylor University (-2)
67. University of Tennessee (-8)
71. Illinois Institute of Technology (+6)
72. Rutgers - Camden (+5)
72. Pepperdine University (-17)
74. University of Richmond (+3)
74. Santa Clara University (+11)
76. University of Oklahoma (-5)
77. Loyola University - Chicago (+10)
77. Georgia State University (-12)
77. Pennsylvania State University (-12)
77. Loyola - Marymount University (-6)
77. Seton Hall University (0)
77. Catholic University (+17)
83. Rutgers - Newark (+4)
83. University of Denver (-6)
83. DePaul University (+4)
86. Seattle University (-9)
86. Marquette University (+1)
88. University of New Mexico (-11)
88. University of South Carolina (-1)
92. Northeastern University (+2)
92. University of Arkansas (+2)
94. University of Louisiana (-19)
94. University of Louisville (+4)
94. University of Nevada - Las Vegas (-19)
97. University of Maine (+3)
97. Hofstra University (+3)
97. Gonzaga University (+3)
100. St. John's University (-13)
101. University of Buffalo - SUNY (-16)
101. University of San Francisco (-3)
From Lawschoolheadlines.com 4/22/09:
"Did Brooklyn Law School Game the U.S. News Rankings?
Today, the Wall Street Journal Law Blog published excerpts from an interview with US News Data czar Bob Morse. In the interview, Morse confirmed that part-time and full-time admissions data were combined in the 2010 edition:
“In the past, we’d just used full-time [admissions data]. But some schools we think were gaming the system. There were some part-time programs that were set up just for US News reporting purposes.”
This new policy is likely to blame for the rankings decline of schools such as Fordham and George Washington. The one school seemingly not impacted by this new policy? Brooklyn. As many of our readers noted, Brooklyn’s part-time program was missing from the new Part-Time Rankings. Our readers further claimed that Brooklyn’s LSAT spread did not match up to the school’s combined program LSAT spread.
Our readers were correct. The LSAT spread found on the US News Rankings (162-165) matches the full-time spread found on the Brooklyn Law website (162-165). It does not look like the part-time spread (158-160) was factored into the new rankings. According to the latest LSAC data sheet, Brooklyn’s full-program LSAT spread should have been 159-164.
Currently, Brooklyn is sharing the 61 spot with a cluster of other schools. A one or two point shift in their overall score would drop the school 5-10 places in the rankings.
So, did Brooklyn game the rankings by not including their part-time program for consideration? Probably. We emailed every dean listed on the Brooklyn website; we will update this post if any respond.'
Good post, but your analysis is based off of last year's rankings (the 2009 rankings). US News just released their 2010 rankings.
A lot of the schools that had the biggest jumps/falls in 2009 had them reversed this year as they "returned to mean."
Another problem is that the voting of the survey takers is not independent of the results from past surveys. ie) The voters know that Alabama was 43 in 2009. They will then rank Alabama relative to where it was in the last known result as opposed to what they really consider Alabama to be ranked (the honest vote).
Thanks for the comment. The "2009" mention in the first paragraph was a typo, if you follow the link, you'll see that shown is a copy of the 2010 ranking. I've corrected this mistake in the blog post. The analysis is based off the newly released 2010 US News Rankings.
You're right about the endogeneity or feedback problem inherent in the surveys, just one more problem US News has to deal with.
If you really want to go nuts about statistical failure, look at the bar exam data. AS you said above, you have to take the difficulty of the bar exam into account. They do that, but only for the state in which the plurality (or majority, if there is one) of the school's students take the bar exam.
Let's say School A has 51% of their students taking the bar in high pass rate state X, and 49% taking the bar in a low pass rate state Y, with 100% passing in both X and Y. School B has 51% taking the bar in Y and 49% in X, with 90% passing in each. School B will be ranked much higher in bar passage rate than School A, because School B will benefit from their 90% being factored with the "hard" bar (with no effect from the easy bar state) and School A will have their 100% discounted for the "easy" bar (with no effect from the hard bar state). School B ranks higher, despite the fact that School B beat school A in both the harder and the easier state.
Not to mention the fact that some states (namely CA) with low pass rates have that not because the bar is harder (though it may be) but because there are many unaccredited schools in those states and/or because they let anyone sit for the bar, whether they went to law school or now.
All US News has to do is to stop looking at pluralities and match each school's pass rate on each state bar up with the pass rate for the state. It's a simple spreadsheet, no harder than calculating anything else. (I doubt there's any way to correct for the second problem, but it wouldn't matter if every school got the benefit from it equally.) But despite the fact that myriad people have pointed this out to US News, they keep ignoring it and refuse to change their methodology.
@Patrick:
I checked again and you are definitely using last year's rankings. I know based on where my law school was ranked.
Here are the ones that were just released:
http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-law-schools/rankings/
They were just uploaded to the US News site. The new ones are the 2010 rankings which are published in 2009.
JEF: Those are the rankings I'm using (if you follow the link in the post, you'll see the photo copy is identical to the ranking now posted on US News website). The list at the end of the post is my recalculated rankings based on the statistics provided by US News in the 2010 ranking. The number in parentheses is the difference between my ranking and US News. It's entirely possible that I may have misnumbered your specific school. I'd happily correct any discrepancies.
Patrick:
The new 2010 rankings at usnews.com are very different from what you used in your analysis. You may need to refresh the US News site to get the latest update.
Compare:
Columbia
US News (2010): Ranked 4
Your site (2009): Tied 3
Minnesota:
US News (2010): Tied 20
Your site (2009): Ranked 22
Alabama:
US News (2010): Tied 30
Your site (2009): Tied 43
Buffalo:
US News (2010): Ranked 85
Your site (2009): Tied 101
JEF,
That is wrong, you need to re-read the post. The list on my website is NOT the 2009 ranking. It is a recalculated ranking from 2010 US News report using the 53% of the data that is not complete nonsense. My site does not reflect the 2009 ranking in any way (Alabama was not 43 in 2009, they were 32; Columbia was ranked 4 in 2009 as well as 2010, I have them tied for 3rd). If you look at my ranking, it says "3. Columbia (+1)". That means if you take the 2010 stats and remove the nonsense measures, Columbia gains 1 spot, going from 4 to 3. Minnesota drops 2 spots going from 20 to 22, Alabama drops 13 spots (30 to 43). Nothing about my ranking reflects any information or the numbers of 2009. The point of this post is to indicate that Alabama (and Buffalo for that matter) are ranked far higher then they should, and US News is a joke (not to offend if you attend either institution). This is by no means a complete or sufficient ranking, but merely a means of pointing out US News inadequacies methodologically. Good rankings that actually involve some original research are out there, such as Leiter's Law Ranking.
I'll reiterate, the ranking on this website is calculated based upon the 53% of data from the 2010 US News ranking that makes sense. If you drop measures that you would have to be literally mentally deficient to use, Minnesota, Alabama, and Buffalo would all fall in the rankings. This means that Minnesota, Alabama, and Buffalo are all increasing their rank through the "bad" measures I discuss in the post. If you use the 53% of the data provided in the US News 2010 ranking that I outline, what I have posted is the resulting ranking. Not the 2009 ranking, the 2010 Ranking.
You apparently attend Minnesota, as that's the only one where my ranking and the 2009 actually correspond. To illustrate:
Columbia
2009: 4
2010: 4
My Site (Recalculated 2010): 3
Minnesota
2009: 22
2010: 20
My Site (Recalculated 2010): 22
Alabama
2009: 32
2010: 30
My Site (Recalculated 2010): 43
I stand corrected. I misread your post and your explanation is quite correct.
I was reading quickly and got distracted once I saw that my law school was ranked lower in your list relative to it's place in the US News list.
I have a new suggestion for you now that I realize that you created your own algorithm. Check out this site:
http://monoborg.law.indiana.edu/LawRank/
It only has data from a few years ago, but makes it easy to rank the schools by re-weighting the various pieces of data. He also added some fun optional data points like Number of Tibetan Restaurants near the school.
I agree, USNews rankings are trash. In no way are they scientifically valid and statistically accurate, yet people in the legal profession love to rely on them (shows how dumb most people are).
Nevertheless, the USNews rankings is in the same category as the tabloids and trash blogs that people can't get enough of.
Patrick,
Thanks for doing this! Would you be willing to share your spreadsheet, to make it easy for interested constituencies from schools outside the top 100 to determine where they would end up based on this methodology? Also, would you be willing to post the "overall scores" generated by your methodology? I'm curious how many schools now ranked between, say, 101 and 120 would rise into the top 100, and how many schools near the bottom of the top 100 would fall out.
Thanks again.
Patrick, where did you get the info that Hastings stayed out of the ranking for a year? I'm not aware that we ever did. Thanks, Nell
Nell,
If my memory serves me correctly, it was either 2005 or 2006. The Hasting's dean called out Bob Morse on multiple occasions to the effect of Morse is an amateur without a phd in anything just trying to sell magazines and is completely unqualified to do statistics. Then Hastings didn't send in a survey and dropped about twenty spots. Feel free to correct me if that's incorrect, but I very clearly remember something along those lines a few years back.