Finding the Wise and the Wisdom in a Crowd: Estimating Underlying Qualities of Reviewers and Items.

The article presents a statistical methodology to process ratings from multiple reviewers, simultaneously estimating unbiased and consistent measures of items' true qualities, as well as individual reviewers' biases and accuracies. This approach improves upon simple averaging by adjusting for systematic over- or under-rating and varying precision among reviewers, demonstrated through Monte Carlo simulations and applied to expert ratings of Bordeaux wines. The method reveals significant variation in expert accuracies and biases, and the estimated wine qualities strongly predict retail prices even when controlling for well-known experts' ratings and other fixed effects. Additionally, the methodology accommodates category-specific biases and accuracies, as shown in the differentiation between left- and right-bank Bordeaux wines, and offers a framework applicable to diverse rating contexts beyond wine, such as consumer products, artistic contests, and peer reviews.