To a substantial extent, what we assess is whether the person who is expressing the ideas in question sounds smart. If you’re going to convince someone to take your perplexing, paradoxical ideas seriously, or if you’re going to convince them that your impenetrable prose is worth the struggle, you had better first convince them that you’re wicked smart. Being good at seeming smart is perhaps the central disciplinary skill for philosophers…
This might explain why no academic discipline is more obsessed with the intelligence of its practitioners than philosophy. The philosopher Sarah-Jane Leslie and her colleagues recently asked faculty, graduate students and postdoctoral fellows from around the country to what extent they thought their discipline required a special aptitude “that just can’t be taught.” Philosophers agreed with such statements more than scholars in any other subject area.
Unfortunately, seeming smart is not a level playing field. In our culture, white men, especially white men from privileged backgrounds, have a large advantage in displaying the superficial features that attract high expectations.
Feminist Philosophers 
Jessica Wilson made (what I thought was) a convincing criticism of this argument at Daily Nous; it’s not that there’s some superficial things that white men do that make them seem smarter; they get evaluated as smarter no matter how they present themselves.
In response to Matt W.: While it’s true that there’s clear evidence that “white men….get evaluated as smarter no matter how they present themselves”, that’s not evidence that there are no ‘superficial things that white men do that make themselves seem smarter’. (I don’t mean to suggest that you are positing an evidential link, I’m only pointing out that there isn’t one.) These two phenomena are not in competition.
It would be good to see solid, independent evidence that there are no general behavioral differences in contexts in which the demographic imbalances are a stark as they are in philosophy. I myself would be quite surprised if there were none. Given the well-established phenomena of stereotype threat, it would be surprising if there were no general (not universal!) differences in high-stakes performance cases in which such threat is in play. And, though I know of no studies on philosophy in particular, there are, for example, studies suggesting that men are more likely to dominate discussions AND to be not seen as dominating them. I would be quite surprised if this weren’t also racial disparities in contexts in which whites are a majority. These are measurable behavioral differences that measurable make an assessment difference.
On a more “gut” level, I would have thought it pretty clear that white male philosophers are often rewarded for aggression, a behavioral tendency that can be dangerous for women and folks of color.