Blinded versus Unblinded Review: A Field Study on the Equity of Peer-Review Processes.

This article reports on a large-scale field experiment comparing single-blind review (where reviewer identities are hidden) and double-blind review (where both reviewer and author identities are hidden) in the context of 530 submissions to the Society for Judgment and Decision Making’s annual conference. The study assessed the two review systems on reliability, fairness, and validity, finding that both exhibited moderate reliability with only about 40% overlap in top-rated submissions, indicating substantial noise in the review process. Single-blind review favored submissions with more senior coauthors and first authors who were PhD students or research scientists, while disadvantaging Asian first authors; male first authors received slightly higher ratings in both conditions, especially under double-blind review. Neither review system’s ratings predicted talk quality, attendance, or audience engagement, but both similarly predicted poster quality and eventual publication, suggesting limited differences in validity. The findings highlight that double-blind review is not a complete solution to inequities present in single-blind review, and that including author identities does not substantially improve reliability or validity but may introduce biases, underscoring the need for evidence-based approaches to designing merit-based evaluation systems.