PT - JOURNAL ARTICLE
AU - Tong, Xin
AU - Feng, Yang
AU - Li, Jingyi Jessica
TI - Neyman-Pearson classification algorithms and NP receiver operating characteristics
AID - 10.1126/sciadv.aao1659
DP - 2018 Feb 01
TA - Science Advances
PG - eaao1659
VI - 4
IP - 2
4099 - http://advances.sciencemag.org/content/4/2/eaao1659.short
4100 - http://advances.sciencemag.org/content/4/2/eaao1659.full
SO - Sci Adv2018 Feb 01; 4
AB - In many binary classification applications, such as disease diagnosis and spam detection, practitioners commonly face the need to limit type I error (that is, the conditional probability of misclassifying a class 0 observation as class 1) so that it remains below a desired threshold. To address this need, the Neyman-Pearson (NP) classification paradigm is a natural choice; it minimizes type II error (that is, the conditional probability of misclassifying a class 1 observation as class 0) while enforcing an upper bound, α, on the type I error. Despite its century-long history in hypothesis testing, the NP paradigm has not been well recognized and implemented in classification schemes. Common practices that directly limit the empirical type I error to no more than α do not satisfy the type I error control objective because the resulting classifiers are likely to have type I errors much larger than α, and the NP paradigm has not been properly implemented in practice. We develop the first umbrella algorithm that implements the NP paradigm for all scoring-type classification methods, such as logistic regression, support vector machines, and random forests. Powered by this algorithm, we propose a novel graphical tool for NP classification methods: NP receiver operating characteristic (NP-ROC) bands motivated by the popular ROC curves. NP-ROC bands will help choose α in a data-adaptive way and compare different NP classifiers. We demonstrate the use and properties of the NP umbrella algorithm and NP-ROC bands, available in the R package nproc, through simulation and real data studies.