Relevance-Based Importance: A Comprehensive Measure of Variable Importance in Prediction.

The notion of variable importance is not uniquely defined. If a prediction is formed from a linear regression model, it is common to measure variable importance as a t-statistic, but a t-statistic is difficult to interpret if the predictive variables are collinear, and it is uninterpretable if the relationship between the predictive variables and the outcomes shifts as conditions change. A Shapley value measures variable importance when a prediction is formed from machine learning models. It is robust to collinearity and conditionality, but it does not account for a variable's contribution to the reliability of individual predictions. It only considers a variable's contribution to the reliability of predictions on average across all predictions. The authors introduce a new measure of variable importance, called relevance-based importance, that, unlike a t-statistic, is robust to collinearity and conditionality and, unlike a Shapley value, accounts for a variable's contribution to the reliability of individual predictions. The authors show that, in the special case in which the predictive variables are uncorrelated with one another and the relationship remains constant, relevance-based importance provides the same information as a t-statistic when averaged across all predictions. They also show that when relevance-based importance is averaged across all predictions, it converges to the Shapley value where the chosen value function is the R2 of a linear regression model. [ABSTRACT FROM AUTHOR]