On the Use of Outcome Tests for Detecting Bias in Decision Making.

This article focuses on the methodological foundations and limitations of outcome tests used to detect bias in decision making by judges, lenders, and other gatekeepers. It frames decision making models as members of the Roy model family, distinguished by how tightly potential outcomes influence decisions, and contrasts the Generalized Roy Model (GRM) with the more restrictive Extended Roy Model (ERM). The authors demonstrate that outcome tests are logically invalid under the flexible GRM—where decision thresholds can vary with unobserved individual characteristics—because differences in marginal outcomes across groups do not reliably indicate bias. In contrast, restricting decision makers' behavior to the ERM, where thresholds vary only by group membership (e.g., race) but not by other unobserved traits, restores the logical validity of outcome tests and enables econometric identification of marginal treatment effects using local instrumental variable methods. The paper further discusses challenges in empirical settings with discrete or endogenous decision makers and highlights necessary assumptions—such as the absence of racial differences in unobserved characteristics or outcome functions—for average outcome comparisons to be informative about bias. Overall, it provides a rigorous framework clarifying when and how outcome tests can validly detect bias, emphasizing the importance of explicit decision models and careful definition of bias for empirical research.