Sunday, November 25, 2007

Leiter on Leiter (and the rankings)

Brian Leiter was kind enough to respond to my post on his latest set of rankings. My posts (here and here) speak for themselves, so I will just respond to a few issues Professor Leiter raises. Leiter defends the rankings by emphasizing his purpose: to inform law students about law schools, and he has rightly suggested that his data may be more illuminating than the rankings supplied by US News and World Report. But of course, once out, his rankings are open to being used by others for various other purposes. This happened right after his listing of the top 10 on each law school faculty, when another blogger (who I will not link to so as not to embarrass him) mentioned himself as among the faculty at his school with the greatest scholarly impact. I suspect many of the top ranked will note their status in annual reports to their Dean about their scholarly progress.

A central concern raised on this blog is about how rankings like this may be used by Deans and by other administrators. Administrators like data. Numbers are great because they purport to be measures of something. If there is a ranking, that can enable a Dean or Provost to gauge how a school or a faculty member is doing based on how high they rank, and whether their rankings go up or down.

Professor Leiter takes issue with my comment that: "Brian Leiter's rankings are not a true measure of 'scholarly impact,'" since after all he was only trying to measure "scholarly impact in legal scholarship." Because Leiter's rankings don't include scholarly impact outside the Westlaw JLR database, there is a risk that incentives will be created by administrators concerned about rankings. My first post on this put it this way:

Once something is seen as "countable," and both law schools and individual faculty are identified based on how their numbers line up...you’ve seen the next step before. Law schools decide they need to move up the rankings. Perhaps they reward faculty based on how far up their numbers are. All of a sudden, that leading, peer-reviewed journal, outside the Westlaw JLR database, is no longer a great prize on your C.V. Your efforts on that were a waste of your law school’s resources.


I pointed out that leading legal historians are not included in the rankings because they do not have full-time appointments in a law school, and I also noted the way the rankings skew in favor of larger sub-fields (like American legal history as compared to medievalists and European or Asian legal history). This point would not matter if Professor Leiter was only ranking schools. Once he turns to rankings within fields, it is important to emphasize the incompleteness of the lists. Appointments committees interested in a major hire in legal history, for example, would miss Christopher Tomlins at the American Bar Association and Henrik Hartog in the History Department at Princeton, and also leading legal historians at law schools in fields other than American legal history. Again, Leiter's lists will be used for purposes other than informing law students, especially rankings of individual faculty members.

Professor Leiter and I appear to differ on the kind of scholarly impact law professors should aspire to. I do think that we all should aspire to be leaders on our campuses not just in our own schools, in the academy not just the legal academy, and in the world not just in the United States.

The comments were not open on his blog, which is why one needs to turn to some unlikely places to start a conversation. Much attention will be paid to Professor Leiter's rankings, so it is important to discuss them. Others have taken up the topic here and here.
Update: Brian Tamanaha weighs in, with a post on the nature of citation practices in U.S. law reviews.
Another update
: The discussion continues here and here. An earlier post I'd missed, raising questions, is here. Building on Leiter's data is a post by Paul Butler on Blackprof, regarding highly cited minority scholars, and Jack Chin.
And more
: Rankings are like crack and we're addicted, says Daniel Solove.

5 comments:

Brian Leiter said...

I'm afraid I don't have the time required to monitor comment sections on a regular basis, which is why I open comments only infrequently. Trackbacks, however, are open, and you should feel free to send one linking to your posting.

Mary L. Dudziak said...

Thanks for the suggestion. Unfortunately Blogger doesn't support trackbacks.

Brian Leiter said...

I didn't realize that. I've added a link at the bottom of my post to this reply.

Mary L. Dudziak said...

Thank you!

Alfred Brophy said...

Thanks to you both for engaging in this important discussion.

One thing that struck me is the gender imbalance revealed by Professor Leiter's data. Nine of the top ten legal historians are male. Partly this is due, I suppose, to the way that Leiter's study preferences people who've been in the business a long time. (Only one of the top ten is under age 50; seven of the ten are above age 65.)

I think that also highlights a problem with the legal history profession. I hope there will be some more discussion here at legal history blog about the gender and racial imbalance in our profession and how that impacts what's studied and who enters the profession.

I also agree that there is a need to break out the historians of the United States from the rest; as Leiter points out, historians of places other than the US are at a serious disadvantage in citation studies (as are those who work on in areas other than constitutional law).