November 12, 2014

UW statistician, philosopher win prize for detecting bias in peer review

In the wake of a 2011 study that found black applicants for National Institutes of Health grants were significantly less likely to receive funding than their equally qualified white counterparts, the health agency began to look at ways to uncover and address bias in how it awards research funding.

The agency launched a contest last spring to detect bias and boost fairness in how it reviews grant applications. The “Most Creative Idea for Detection of Bias in Peer Review,” went to an idea proposed by Carole Lee, a UW assistant professor of philosophy who specializes in peer review, and Elena Erosheva, a UW associate professor of statistics and of social work.

They proposed to identify “commensuration bias,” or the idea that bias can creep in when evaluators combine several ratings in different categories to come up with an overall score. They proposed to see whether black and white investigators receiving comparable scores in the different categories (significance, innovation, investigator, environment) ended up with significantly different overall impact scores.

“Research in social psychology suggests that, when evaluating job applicants along multiple criteria (like education and experience), evaluators prioritize whichever criterion favors the in-group applicant (white/male) versus the out-group (black/female) applicant, which has the effect of boosting the white/male applicant’s overall score,” Lee explained. “Analogously, we hypothesized that at NIH, white grant applicants receive higher overall impact scores than minority applicants in cases where they have received identical (or sufficiently similar) scores on sub-criteria.”

Judges at the NIH Center for Science Review were impressed with the uniqueness of the UW pair’s approach. Their submission provided methods to flag proposals where the overall scores didn’t match up with individual category scores, prompting program officers to look further into the review process.

The first-place prize is $10,000, and the UW researchers hope to put their idea into practice.