< Back to previous page

Project

Probability of Improved Prediction: a new concept in statistical inference (R-11615)

In many empirical sciences there is a consensus that the Scientific Method can rely on statistical hypothesis testing, and that the p-value can be compared to a threshold of 5% to come to a binary decision: either reject the null hypothesis (a "positive" result) or accept the null hypothesis (inconclusive). The former leads easier to publication, whereas many journals are hesitant publishing negative results. This tradition leads to publication bias. Moreover, sometimes researchers do significance-fishing, which results in an increase of the false positive rate and in irreproducible results. Many scientists and statistician are criticising this use of p-values nowadays. We propose an alternative summary of experimental data that may overcome some of the issues related to p-values. We believe that our statistic is relevant to the Scientific Method, because it has an interpretation in terms of predicability of a model. In this sense, it may help to connect statistical science with predictive modelling or machine learning. We propose the Probability of Improved Prediction (PIP), which measures how more often a model gives better predictions than another model (the two models may differ in e.g. a single predictor). We study the basic properties of the PIP and its relation to the p-value for several traditional statistical models. We also propose and study several estimators of the PIP, and study its use in model selection procedures for prediction models.
Date:1 Jan 2021 →  Today
Keywords:p-values, prediction, Statistical inference
Disciplines:Statistics