There’s some really interesting discussion starting to take place about the pros and cons of various kinds of ranking systems for philosophy. A couple of years ago I wrote a paper on the potential for implicit bias in both the PGR and the REF. I thought it might be useful for that paper to be a part of these discussions, so I’m posting the penultimate draft of it here. (Also, I can’t figure out how to use my university’s newly updated CMS. Grr!)
The paper is “Ranking Exercises in Philosophy and Implicit Bias”, in Journal of Social Philosophy, 43:3, 2012.
It was only after publishing the paper that I noticed another interesting difference between REF and PGR. I’m no great fan of the REF– it has lots of problems, but it does have the nice feature of not weighting an areas of philosophy more heavily than any others. Whatever area your work is in, it’s only ranked by people in your area, and there’s no overall ranking of departments, except in so far as various competing ones can be (and are) arrived at through the rankings of work, impact, etc. So there’s no case to be made that a department will do better in the REF by hiring an analytic metaphysician than a pragmatist. Departments are free to just go by quality and teaching/supervising needs, without worrying that they should favour particular areas for the sake of the rankings.
Feminist Philosophers 
I stumbled upon essentially the same sort of argument independently a few years ago but, unlike you, was too lazy to do anything with it, other than mention it in emails to people and the other day over at Weatherson’s blog (http://brianweatherson.tumblr.com/post/98362665004/after-the-pgr#comment-1605302426). It seems to me a very strong argument: (i) PGR is a “reputational survey”; (ii) Reputation (as you and others have made extremely clear) is a gendered (etc) commodity; (iii) It follows, or sure seems to me to follow, that the PGR results can’t but be gender (etc) biased; (iv) Since PGR is not just a measure but a source of reputation, we have a feedback loop, and so PGR not only reflects but reinforces gender (etc) bias.
Of course, just to be very clear, that doesn’t mean that anyone involved with PGR is *trying* to rank in a way that is biased or to reinforce such biases. But once it is known that the rankings have this sort of effect, that would seem to generate some kind of duty.
Strictly speaking you are right that the difference between REF and PGR you mention in the last paragraph of your post should exist. However, it is not clear to me that this is quite so. I think that judgments about the relative importance of different areas of philosophy are widespread in the UK too and may well influence REF readers who are forced to read tons of stuff (and not exclusively in their area). It may be relevant to note that
since the introduction of REF the number of primarily non-analytic or mainstream departments in the UK has shrunk. I do not think this is a coincidence since REF readers have been mostly analytic so that the continentalists on the panel would have had to read everything French, German, Italian, aesthetic, metaphysics, what’s not. To see how bizarre this is, one should consider what we would think had it been the other way round.
Plus there is an overall ranking of departments at the end. It might not be official, it might not correlate with money one gets but every vice-chancellor treats the GPA as offering a ranking. This is not to deny that the REF has also had beneficial consequences.
Yes, you’ve got a point. I do know that there are plenty of departments that *believe* they must hire in M&E to do well in the REF. However, at Sheffield we have long had panel members who specialise in Pragmatism and Hegel, so we’ve never had this false belief. They are very insistent that the belief is false, and their evidence base is good. But the false belief if widespread can still do enormous harm as you note.
But the GPA thing has become more complicated, as there are now so many different ways to calculate.
I don’t think REF is generally great– just that it’s better to have specialists ranking individual areas than non-specialists making overall judgments. (That’s the reason that I argue in the paper that the PGR would be much better if it dropped the overall rankings.)
Hey Richard,
An interesting problem that your point raises is that most alternations are likely to be gendered and some are less susceptible to undermining and discounting. For example, citation counts are really quite mesmerising to people and we know how messed up they are in so many ways.
A big problem with all the rankings is the use to which they are put. They are extremely high stakes, thus encourage all sorts of distortion of behavior. What’s more, since the stakes (in the case of REF a big chunk of funding plus better shots at other funding) directly support both the direct and indirect properties measured, they increase the barriers to change.
I’m certainly not wanting to give a big defense of the REF!
but it does have the nice feature of not weighting an areas of philosophy more heavily than any others.
This may well be a good feature of the REF. (I’m no expert on it, but my understanding is that it seeks to measure the quality of research a department puts out. So long as we assume that an areas of research should be pursued at all [not _always_ the case, but usually, I think, in most fields] it makes sense to measure that research against others in terms of quality only.) But, this seems to me to just be one way that the PGR and the REF have different goals, and so are not comparable directly. Is it really the case that people here think that _no_ issues in philosophy are _more_ “central” than any other? (I’m pretty sure that Richard Heck doesn’t think that, for example.) It seems pretty plausible to me that, to be a really well trained philosopher, you need at least some training in epistemology, metaphysics, and logic, as well (I’d argue) the history of philosophy, including it’s most “important” areas, set out in terms of area of influence. (Not intrinsic importance, but for understanding the later developments.) If this is right (I’m not _certain_ of it, of course, but it seems plausible, and I think the practice of most people indicates they accept something like it, too) then it makes some serious sense to give a rankings “hit” to departments that have deficiencies in the areas that we think all good philosophers need training in. Does this mean that these areas are set for all time? No, it doesn’t. (I think that common perception about the importance of philosophy of language, for example, has changed, largely for the better, in my impression.) Does it mean that a grad student would always be wrong to choose a program that had fewer good philosophers working in these areas? No- that would depend on interests, over-all options, idiosyncratic factors, and so on. But, it does suggest that, _for certain purposes_(*) some areas of philosophy are more “central” and so worth weighing somewhat more heavily than others.
(*) Many critics of the PGR, though of course not all, forget that it’s not meant to be, and doesn’t claim to be, a _general_ measure of philosophical value or importance, but rather to be a guide for choosing grad programs. For _this_ purpose, weighing different areas more heavily than others seems right to me, for reasons like those set out above, even if it would not be right for the completely different purposes of the REF.
Matt, whether or not there are more central areas of philosophy, I think we’re so bad at detecting them that we should be a little nervous about giving our guesses much weight.
Here’s a true story to illustrate this. When I was in grad school I thought basically the central question in philosophy was the nature of mental content. This wasn’t a self-interested thought: I don’t work on mental content, and wasn’t any more interested in it than in any other area of philosophy. But given its importance both to Modern philosophy (at least as it was then taught) and to the highest profile debates of the 80s and 90s, it seemed to me simply true that mental content was one of the central philosophical questions.
Twenty years later, that doesn’t seem so true. The stuff that I did start working on in grad school, how to formally model rationality, and how the nature of those models affects debates in philosophy and social science, seems to be somewhat more central. It’s now what people call epistemology! But I suspect this is a passing trend too.
The central questions in philosophy do mostly seem to revolve around the nature of values, minds and the world. But that’s such a broad claim that it captures just about everything in philosophy, and much of the rest of academia. I’m a little sceptical that claims more precise really do capture anything more than fleeting trends.
Here’s another reason for worrying about centrality. Lots of things become peripheral to philosophy because they become part of another field. Lots of questions that were central to philosophy 150 years ago are now simply part of psychology or economics, for example.
Some days I think that’s the only way things become central. So centrality is a measure not of how important something is philosophically, but of how unimportant it is non-philosophically. So understood, it isn’t something we should reward I think, even if giving advice for grad school.
I’m sympathetic to some of that, Brian. But I’m curious. If a grad student in your department told you that they didn’t think they should have to take _any_ epistemology, metaphysics, or logic, would you agree with them? Have you argued that there should be _no_ distribution requirements in the departments you have been involved in? Those are real, not rhetorical questions. I don’t have any deep stake in this. I’m curious to know if people think it.
Also, I think it’s important to distinguish things which are central because they are part of basic competence and things which are central because they are the focus of current research. These are not necessarily in alignment. So we can agree to Matt’s list even as described and also endorse a listing of topics central to research which would look totally different. Arguably, for grad school, you want at least competence in the former but excellence (in some sense) in the latter.
And this highlights Brian’s points: Research is subject to trends and fashion. Exacerbating sensitivity to such may be highly counterproductive.
I would say that about logic. I’m actually thinking of trying to have our department move from having a logic requirement to having a more general ‘formal methods’ requirement. Someone who can do grad level game theory work doesn’t also have to learn whatever would be taught in a logic course.
I’m a bit biased though. I came from a program that had no graduate coursework, and hence no distribution requirements, and I turned out OK. (Biased assessment alert!) We just went to whatever seminars/colloquia etc we felt like going to, and picked up stuff that way. So I’m really not wedded to distribution requirements at all.
More controversially, I really don’t think metaphysics is particularly central to philosophy. If one hadn’t followed any of the debates in metaphysics in the last twenty years, would that impact at all one’s ability to write about contemporary work on moral worth, or peer disagreement, or relativism? I don’t really see it. A central topic should be one that you’ll seriously lose out on elsewhere if you don’t follow it, and I don’t see metaphysics having that status. By this measure I suppose feminist philosophy is more central than metaphysics, but that’s going to be a somewhat contentious judgment.
Those areas would be central only within analytic philosophy, though, right? In continental philosophy what you need to know about most is certain figures deemed central, whose work for better or worse figures into a great deal of subsequent discussion. Typically – Hegel, Heidegger, Nietzsche – their thought cuts across epistemology, ethics, etc.
I totally agree with Alessandra that since the RAE/REF arrived the extent of non-analytic philosophy in the UK has shrunk. Also the extent of ‘applied’ philosophy, call it what we will. But I think the problem (now) is people’s perception of what areas will do best in the REF rather than what necessarily will. Or alternatively, the REF can in some cases serve to reinforce people’s prejudices. It does seem that the view that e.g. analytic metaphysics is best for the REF lacks solid evidence. In the last RAE for instance Dundee did quite ‘well’, ditto Middlesex, ditto Manchester Metropolitan (all broadly continental focused). Yet I’ve had conversations with people in which they just treat these cases as odd anomalies to set aside or remark that such outcomes are weird!
More controversially, I really don’t think metaphysics is particularly central to philosophy. If one hadn’t followed any of the debates in metaphysics in the last twenty years, would that impact at all one’s ability to write about contemporary work on moral worth, or peer disagreement, or relativism?
This might indicate one way that it’s easy not to understand each other (or talk past each other) on these things, I think. My own view is that a good amount of contemporary analytic metaphysics (“pre-critical metaphysics, I might say) is a waste of time. But, I’d say that learning how to think about metaphysics is very useful for all philosophers, and that, for example, lots of work on ethics (or sometimes legal philosophy) is made much worse by making pretty basic mistakes that wouldn’t be made, or made so easily, if the people doing the work knew a bit more about metaphysics. (A lot of the best commentaries and criticisms of Michael Moore’s recent book on causation in the law, for example, have come from people working outside philosophy of law, working in metaphysics, for example.) To this extent, what I think is important is not knowing what’s hot, or the newest trends, but learning _how thinking about_ metaphysics or epistemology or logic, or formal thinking more generally. Learning some examples is one of the best ways to do that, I think.
As for “continental” philosophy, I’m not an expert in any of it, though some parts of it are significantly relevant for my work and so of interest to me. Even there, it seems that one can’t know what, say, Hegel, Heidegger, or Nietzsche is up to w/o knowing a good deal of metaphysics, epistemology, and other topics. Heidegger, after all, wasn’t doing “Heidegger studies” (at least not when what he was doing was good!) but rather, metaphysics and action theory, and things like that. (Again, it often seems like what’s wrong with _bad_ work on, say, Nietzsche is that it makes basic mistakes about metaphysics and epistemology [confusing the two, often enough] in a way that suggests that studying those things would be very useful to someone wanting to work on Nietzsche. That doesn’t of course, have to mean studying contemporary analytic epistemology in any of its variations.)
(I should add, finally, the I think I agree with at least most of what Bijan says above.)
At Sheffield, we have no distribution requirements at UG or PG level. Never have had them, in the 19 years I’ve been here. And our students turn out wonderfully well.
That’s interesting to know, Jenny, and a useful data point (though following Brian W above, I’d wonder if the last bit is slightly ironic in a thread on implicit bias -though perhaps the evaluation of one’s own students is more of the “explicit” sort!)
Yes, of course, anyone’s evaluation of their own students should be taken with loads of cautions.
Well, I think it’s interesting that the UK has basically none of the diversity/core/well roundedness stuff at either the undergraduate or graduate level and yet doesn’t seem to put out substantively worse graduates in any obvious, aggregate way.
Believe me, I went in to my UK job horribly confused at how it was remotely possible to give anyone a decent education within the constraints available to us (3 year undergrad, 1 year MSc, 3/4 PhD with no/few classes). But…it’s possible.
(We in the CS wonder how life sciences get 80% success rates in the PhD and are told, “Well, they are more directive” and we piss and moan about how it’s better to let people explore their interests. And then I think, wait, it’s not better for the people who don’t complete.)
I’d really love some solid studies to look at. But prima facie I think people advocating extremely expensive educational plans have a bigger burden.
(I’ve had people just assert that UK PhDs aren’t as good. C’mon.)
I think we should be SUPER careful about ineffable benefits. I value lots of the diversity and freedom I had (sorta). My senior silversmithing class? Awesome. Latin, French, and Logic…super.
But…were they necessary to a BA in philosophy? To a BA? To going to grad school? To spending over a decade finishing my PhD (whoops…that was a different set of issues :)).
I like what Brian say6s about ‘central areas’ a lot. I would take it a step further and say that it seems to me that the most original work in philosophy – the kind that breaks open paradigms, and reshapes the field for future generations of work – often comes from some kind of pluralist methodological training, from which the most creative kind of philosopher blends ideas and theories in powerful new ways. Given that, for quite some time I have been despairing of a methodological narrowing that it seems to me has been encouraged in our field by TPG and what seem to me to be its clear biases towards certain kinds of analytic philosophy – most insidiously expressed in indirect ways such as reporting which hot new ‘star’ has been hired where (And haven’t so many souls dreamed of being described in such terms, up to the present day? And haven’t so many others felt depressed and worthless when looking on at this king-making process and realizing there is no way in hell – etc.? The philosophical-career-building equivalent of reading fashion magazines.)
How would the next Wittgenstein fare in our current system? I suppose if he (or she!) had Russell writing an undergrad reference, he (or she) would be likely to get into Princeton. But what then?
Bijan– As far asI know we’re the nay UK programme with no core modules at UG level. Not so sure about PG.
Cathy– agreed.
Also worth noting: Suppose it’s right pedagogically it’s important to for all students to take some particular subjects, e.g. logic. What that would mean is that to do a decent job every department must have a good teacher of that subject. It doesn’t at all suggest that research in that area should be viewed as more important than research in other areas.
Whoops. Yes, at the UG level at Manchester in CS we have a lot of structure. PGT there’s no course (other than the project related courses) that is taken by everyone.
So it’s more of an analogy. People make the same sort of noises about the pedogogic benefits of a liberal arts education and for a long, 2-3 years of classes PhD. I think we need to scrutinize such claims very carefully.
I totally agree that there needn’t be a connect between the basics and what’s fruitful to research.
Sections 3.1 and 3.2 of Jenny’s paper, and Richard’s elegant summary of what I think is the key argument there, capture exactly what has always seemed worrying about the PGR. Thanks v much to both.
Thanks for flagging your excellent paper to attention, Jenny.