Patrick: April 2009 Archives

(Author's Note: This post originally only included the top 50, but has now been expanded to include the entire top 100)

Leaked recently are copies of the 2010 US News law school rankings, providing yet another example of why journalists should not be charged with composing an index.  US News collects data on quality assessment (survey data), selectivity, placement success, and faculty resources, standardizes each measure and weights it appropriately to create a final score ranging from 1 to 100.  Obviously, any ranking will have it's shortcomings, but US News has done a spectacular job of creating a ranking which is nothing short of total and complete nonsense.  Before diving into criticism, a disclaimer: I am not a law student nor am I a journalist, I am a political science PhD student who deals with issues of measurement and quantification regularly.  Additionally, I have a master's degree in political science with a minor field in methodology.  The ranking replacement I provide at the end of this post does not feign to be in any way adequate, but is far more accurate a portrayal of these institution's quality than that provided by the magazine.

1. Quality Assessment - This may be the most potentially relevant measure US News includes, as what matters more than the perception of your potential future colleagues and employers?  Some schools will claim particular biases within the data against certain regions (an east-coast bias or a bias against the south-east).  Given that US News claims to be a national ranking, those so-called "biases" are not relevant.  Unfortunately, US News does not tell us how the survey was scored, what weights were used, clustering, etc.  It is not unlikely that they simply took the raw average result without applying any relevant weights, leading to potentially biased results (particularly given the 31% response rate among professionals).  Furthermore, US News does not tell us the number of respondents nor the variation in responses, all necessary pieces of information to determine the accuracy of their scores.
2. Selectivity - First, "GPA" -or- "how to use a measure that is completely irrelevant".  The US News data is measured by median undergrad GPA, comprising 12.5% of the entire ranking.  However, no controls exist in measuring undergrad GPA across different institutions.  US News is assuming that a 3.9 from the University of Mississippi is equivalent to a 3.9 out of Yale.  Please, you're kidding right?  Second, they include acceptance rates.  Unfortunately, these are also difficult to compare across schools given differences in class sizes.  Smaller schools will inherently have lower acceptance rates without regard to whether those students are provided a better education.  This measure would be relevant only if the rankings controlled for institutional size.
   
3. Placement Success - Again, also not relevant.  All jobs are not created equal, and a job with a firm in Beaumont, Texas is slightly different from one in Lower Manhattan.  If this glaring oversight were not enough the reported employment rates are not specific to law jobs.  A school may count their student employed whether they are working for a large firm, a judge, Starbucks, or the local meat packing plant.  I would hope students are actually interested in getting law jobs after 3 years of school and a truck load of debt.  Second, the most irrelevant of all measures: Bar Passage Rate.  Like GPA, this is simply not comparable.  You cannot equate passing the Wisconsin Bar to passing the California Bar, the Wyoming Bar as identical to the New York Bar, or the Louisiana Bar as no different from the Virginia Bar.  If passage rate is to be included it must be weighted by the state bar being taken.
4. Faculty Resources - A potentially helpful measure, but most of these statistics US News does not release in the issue.  It includes financial aid, expenditures per student on instruction, library services, library volumes, etc.

What is to be done?  Well a variety of items, such as number of clerkships, salary after graduation, quality of focus job market (New York>Indianapolis), etc. should be included.  I, however, am not interested in contributing my time to further research, but will leave data collection to lawyers and journalists.  If we use the numbers provided by US News that possess any potential relevancy and continue the use of their arbitrary weightings, the resulting ranking is quite different than that published.  For our purposes we will assume that the survey responses are relatively accurate and include both peer (25%) and professional (15%) assessments.  In the category of selectivity, the only standardized measure we can compare across schools is median LSAT score (10%).  Finally, I'll also include student/faculty ratio (3%), which is unfortunately the only element of the faculty resources category available.  Due to the impossibility of comparing across institutions in their current form, undergrad GPA, acceptance rate, employment rate, and bar passage rate are all excluded (honestly, you'd have to have the IQ of a donut to think they were comparable). 

What's left is the 53% of the initial US News ranking that is provided and does not exist in the realm of total absurdity. The ranking is listed below.  Notably, the top 15 remains relatively stable (no one enters or leaves, though UCLA does step down from 15 to 16).  The further you get from number 1, however, the more innacurate US News becomes.  Within the top 50, three schools are over or under ranked by 10 or more.  UC-Hastings should be 10 ranks above where it currently is, but that is the price they pay for boycotting US News one year.  Indiana is overranked by 10, but I'm sure the US News' claims that they are tied with Illinois will come as a shock to most Midwesterners. 

Finally, the greatest disparity of any school between the below rankings and those of US News top 50 is Alabama.  Alabama provides a valuable example of why the US News rankings are so flawed.  Based on any sensible measure, Alabama should be ranked in the 40's.  So why the sudden jump to 30?  Why is number thirty having such a hard time getting their students law related jobs? Looking at the data, Alabama achieves this incredible accomplishment by gaming the absurdities of the US News system (which everyone knows they've been doing for some time in attempts to improve their reputation, which is number 43 in the survey).  To decrease their acceptance rate, they've begun offering prizes to students who apply, such as IPods.  Clearly this will increase your application rate and correspondingly decrease your acceptance rate, while maintaining absolutely no relevance to the quality of your institution.  Secondly, Alabama does not attract students from as equally difficult undergraduate institutions as many of the competing schools throughout the thirties and forties, unless you actually believe that an undergrad at Alabama is equal to an undergrad at top tier Georgia or North Carolina.  Dream On.  To make matters worse, Alabama Law has announced that it will accept students from the University of Alabama Honors College without requiring an LSAT.  Why would they do such a thing?  Obviously, despite their high GPA's, a student at Alabama Honors College isn't going to do well on the LSAT because, frankly, they're not that smart.  If they were, they would have gone to Auburn.  Furthermore, various rumors have circulated regarding Alabama's improperly reporting student's final GPA's upon completion of undergraduate.

The absurd over-ranking of Alabama should come as no surprise, they are experts at gaming the system no matter the area (see BCS 2008).  However, eventually you get caught and embarrassed, just like losing to Utah in the Sugar Bowl.  The comparison between the law school rankings and the BCS is a good one, as both rankings are completely absurd and a failed measure of actual quality.

Biggest Gainers:
1.  (+21) - University of Oregon
2.  (+17) - Catholic University
3.  (+13) - Temple University and University of Pittsburgh

Biggest Losers:
1.  (-19) - Louisiana State University and University of Nevada - Las Vegas
2.  (-17) - Pepperdine University
3.  (-16) - University of Buffalo - SUNY

The 53% Ranking (Difference Between Ranking and US News)
1.  Yale (0)
2.  Harvard (0)
3.  Stanford (0)
3.  Columbia (+1)
5.  Chicago (+1)
6.  New York University (-1)
7.  University of Virginia (+3)
8.  University of California - Berkeley (-2)
8.  University of Michigan (+1)
10.  University of Pennsylvania (-1)
11.  Duke University (-1)
11.  Georgetown University (+3)
13.  Cornell University (0)
14.  Northwestern University (-4)
15.  University of Texas (0)
16.  University of California - Los Angeles (-1)
16.  Vanderbilt University (+1)
18.  Washington University - St. Louis (+1)
18.  University of Southern California (0)
18.  Emory University (+2)
21.  George Washington University (+7)
22.  University of Minnesota (-2)
23.  Boston University (-3)
23.  University of North Carolina (+7)
23.  University of Notre Dame (0)
26.  Boston College (0)
27.  Washington and Lee (+3)
27.  University of Illinois (-4)
29.  University of California - Hastings (+10)
29.  University of Iowa (-3)
29.  University of Wisconsin (+6)
32.  College of William and Mary  (-4)
33.  Ohio State University (+2)
33.  University of Indiana (-10)
35.  University of California - Davis (0)
35.  Wake Forest University (+5)
35.  Fordham University (-5)
38.  University of Washington (-8)
39.  University of Arizona (+4)
39.  Tulane University (+6)
41.  University of Georgia (-6)
42.  University of Colorado (+3)
43.  University of Alabama (-13)
43.  Brigham Young University (-2)
43.  American University (+2)
46.  University of Maryland (-3)
47.  University of Florida (+4)
47.  George Mason University (-6)
49.  Arizona State University (+6)
50.  University of Utah (-5)
51.  University of Connecticut (+1)
52.  Case Western Reserve University (+3)
52.  Temple University (+13)
52.  Southern Methodist University (-3)
52.  Yeshiva University (-3)
56.  Villanova University (+5)
56.  University of Oregon (+21)
58.  University of San Diego (+3)
58.  University of Pittsburgh (+13)
60.  University of Kansas (+5)
61.  University of Miami (+10)
61.  University of Houston (-2)
61.  Florida State University (-9)
61.  Lewis and Clark College (0)
61.  University of Missouri (+4)
61.  University of Cincinnati (-9)
67.  Brooklyn Law School (-6)
67.  University of Kentucky (-12)
67.  Baylor University (-2)
67.  University of Tennessee (-8)
71.  Illinois Institute of Technology (+6)
72.  Rutgers - Camden (+5)
72.  Pepperdine University (-17)
74.  University of Richmond (+3)
74.  Santa Clara University (+11)
76.  University of Oklahoma (-5)
77.  Loyola University - Chicago (+10)
77.  Georgia State University (-12)
77.  Pennsylvania State University (-12)
77.  Loyola - Marymount University (-6)
77.  Seton Hall University (0)
77.  Catholic University (+17)
83.  Rutgers - Newark (+4)
83.  University of Denver (-6)
83.  DePaul University (+4)
86.  Seattle University (-9)
86.  Marquette University (+1)
88.  University of New Mexico (-11)
88.  University of South Carolina (-1)
92.  Northeastern University (+2)
92.  University of Arkansas (+2)
94.  University of Louisiana (-19)
94.  University of Louisville (+4)
94.  University of Nevada - Las Vegas (-19)
97.  University of Maine (+3)
97.  Hofstra University (+3)
97.  Gonzaga University (+3)
100.  St. John's University (-13)
101.  University of Buffalo - SUNY (-16)
101.  University of San Francisco (-3)