Comments

Thank you for all the comments. Instead of publishing and answering them individually, I have decided to prepare a Q & A post, where I will address questions and comments as they come up. I will try to update the post as often as necessary. However, I will most probably post updates on weekends only (and certainly not every weekend).

Mistakes and updates

Please let me know of any possible mistakes. Entering a lot of data into a spreadsheet quickly (no, I did not spend months on this...) produces mistakes. I have received some comments about possible mistakes, but after verifying I have only found one clear mistake: Jeff Yates was an associate professor in early 2006 and he should not be included. I will update the spreadsheet in the future, when more mistakes are brought to my attention.

I do repeat a warning on subfield designations: don’t rely on them, they are inaccurate (see also below why)! I don’t have the time and competence to change them, though.

Q & A on possible mistakes (based on actual comments)

Dr. [A] at [University B] should be included

Associate professors, tenured on untenured, are not included here. The CV on the particular person’s webpage says that the person has been an associate professor for a couple of years

Dr. [A] at [University B] has tenure [and should not be included]

If I could, I would certainly include tenure information (there are tenured assistants and non-tenured associates), but I just don’t have information on this. The CV on the particular person’s webpage, updated even after the data collection was completed, indicates that the person is still an assistant professor.

The database includes people who have tenure

See previous question. Also, the database is current as of early 2006, and surely a lot of tenure decisions have been made since then. Similarly, people might have moved since early 2006 (and certainly, many will have moved by September).

I did not receive my PhD from where the database indicates, and I have more publications

This is a tough one. These kinds of mistakes are the most embarrassing, and may hurt the particular person. However, there is nothing I can do until I receive more specific information. As regards the publications, the errors are most likely when the person did not have an updated CV on internet in early 2006. Also, please bear in mind that Web of Science is not perfect (in fact, in one case I noticed that Web of Science did not list a person as an author when that person clearly co-authored the piece - the database corresponds to the real authorship). Also, publication information is based on information as of early 2006, thus all later publications are omitted. If I end up updating the data in early 2007 (or early summer of 2007 when it makes more sense), they will be included then.

The reliability and validity of the data?

This is a good comment. I should have given a general indication of the safeguards undertaken.

Validity should not be a huge problem for this dataset. There are no measures of particularly indefinite concepts. Reliability is a problem, of course. And my safeguards were probably not sufficient for clearing publication standards in, let’s say, top political science journals. I did take a second look at several randomly chosen assistant professors, but that was all. Confidence in the reliability of the data should increase over time, though (especially the more downloads there are, the more people look at it, and the more comments I get).

Publications data

Are forthcoming publications taken into account?

No. This information is simply not available.

People usually count forthcoming publications when calculating their average yearly output, thus Pub/YearwPhD is inaccurate

True to some extent. Will this problem hurt anyone? Hardly, as long as hiring committees look at least briefly at individual CVs.

Also, having individuals’ Pub/YearwPhD right does not seem very important. A more interesting question seems to be the average output of a median assistant professor, especially of a median assistant professor about to go up for tenure. Considering that these assistants are out of graduate school for several years, the forthcoming publications should not increase their average output too much (one "fortchoming" for a person out for five years increases the average by only 0.2). And of course, if we wanted to know the average output of someone out for even a longer time, the forthcoming articles will not matter much at all.

Top 10 rankings

I am not a big fan of individual rankings; I hesitated to put them up. I repeat: please don’t take them too seriously. I believed they would spark some discussion (which they did on some other blogs). And they do provide some interesting information. Also, such rankings are very easy to construct for any other data items (“data” -> “sort” etc…).

Publications (especially when ranking individuals) should not have equal weight

This is a very good comment, and I completely agree with it. The million dollar question is, of course, what the appropriate weights should be.

There are several options. The first is the Hix system (using journal impact factor, see his department rankings. I decided against using this approach (but anyone could try, it just requires extra coding!). First, relying on journal impact factor to rank individuals can be very misleading (I don't have a good link right now, but just enter "journal impact factor" in some search engine, e.g. Web of Science itself, and there are several articles that point out the problems). And I did not want to drop publications in other fields such as economics or psychology, but including them creates problems as impact factors may not be comparable between different fields.

Second option is just giving more weight to some very selected publications, especially the “top 3.” How much then? Triple for APSR and double for AJPS and JOP? Based on what? And, is an IR article in JOP really “worth” twice as much as an IR article in IO?

I just decided to have one raw “top” ranking; anyone could play around with the data and create their own top rankings. The easiest “custom” ranking is “data” -> “sort” -> “NoTop3/*”; I have done the calculations for this.

Break the rankings up by subfield

Again, an excellent comment. However, I don’t feel competent enough to judge everyone’s subfield.

I also want to give yet another very strong warning (repeated from above, and the notes on the data): if you want to get a list of scholars in a particular subfield, don’t rely on the subfield designations. They are unfortunately inaccurate and inconsistent. For example, americanists may be identified as “Behavior,” “Formal,” “Judicial,” “Law,” “Methods,” “PA,” “Policy,” “Positive Political Theory,” “Political Economy” or even something else. The departmental webpages and individual CVs do not give clear-cut designations. I did not want to leave subfield designation completely out, because it may be helpful when looking at a particular person. But again, don’t rely on them!!

Again, thank you for all the comments, and hopefully you find the data interesting.