GRASP: a goodness-of-fit test for classification learning.

The article focuses on developing a novel statistical framework, called GRASP (Goodness-of-fit with Randomisation and Scoring Procedure), for assessing the goodness-of-fit of general binary classifiers to the true conditional distribution of labels given features. GRASP operates without parametric assumptions and only requires query access to the classifier, providing finite-sample type I error control in a distribution-free setting and enhanced power in a model-X setting where the feature distribution is known or well approximated. The methodology formulates goodness-of-fit assessment as a tolerance hypothesis test based on f-divergence measures between the true and estimated conditional label distributions, and employs a randomized scoring and labeling procedure to construct test statistics. Extensive numerical experiments demonstrate that GRASP controls type I error effectively and achieves high power in detecting model misspecification, with the model-X variant outperforming the distribution-free version by leveraging knowledge of the feature distribution; the paper also discusses optimal score functions, including a generative adversarial network (GAN)-based approach, and provides convex optimization algorithms for implementing the test.