ABSTRACT The broad and long-term objective of our research inquiry is to develop receiver operating characteristic (ROC) analysis towards a broadly applicable, practical, accurate, precise, efficient, and user-friendly method for evaluation of diagnostic performance in medical imaging and beyond. The objective of this project is to develop an innovative weighted ROC (WROC) analysis. The central hypothesis is that, by introducing a case weighting factor, WROC analysis can mitigate and eliminate bias in ROC analysis from non-random case samples and infer clinical performance without bias. Specific Aims are: (1) develop WROC algorithms and share analysis software with the research community; (2) develop and validate three WROC analysis applications; and (3) investigate with WROC analysis pivotal ROC study inference bias from non-random case samples. Research design, based on contemporary ROC methodologies and preliminary studies, will be to expand the basic ROC theory by introducing a weight factor to every case, and to develop WROC estimation algorithms for common ROC models including the non-parametric, conventional binormal, and proper binormal models, to develop new WROC software, which will supersede existing ROC software, and to make the new software available to the research community by developing an open, easy-to-use, feature-rich, and publication-friendly online calculator. New WROC algorithms will be used to develop three new applications: to compare meaningfully ROC studies of non-random and non-identical case samples, to design ROC studies with stratified case samples and then apply WROC analysis to model case sample distributions to match random sampling and infer random-sample ROC performance without bias, and to estimate aggregate ROC performance of multiple readers by averaging individual-reader ROC curves weighted by clinical case volume. Finally, WROC analysis will be used to investigate bias in the inference to clinical performance from multi- reader multi-case (MRMC) studies of non-random case samples and WROC analysis as a means to overcome this bias. Methods to be used include mathematical derivation of maximum-likelihood estimations, software development, and validation with Monte Calo simulations. The proposed WROC analysis is premised on weighing cases unequally. This simple addition of a case weight will add a new dimension to ROC analysis. Practical benefits include added analysis flexibility, improved clinical performance inference from laboratory studies, and new ways to design better reader studies. The importance and health relatedness of this research is that once developed, validated, and made available to and used by the research community, the new development will be one step that advances ROC analysis towards a broadly applicable, practical, accurate, precise, efficient, and user-friendly method for diagnostic performance evaluation.

NARRATIVE (RELEVANCE) Receiver operating characteristic (ROC) analysis is an important methodology and cornerstone of medical imaging because it is the main tool for evaluating diagnostic performance of radiologists, medical devices, and new technologies in binary medical decision-making tasks (e.g., diagnosis of a patient as having a cancer vs. not). This proposal, which we call weighted ROC (WROC) analysis, innovates on basic ROC theory and practical applications alike by introducing a new case-weight factor, which will add flexibility to ROC analysis, improve clinical performance inference from laboratory studies, produce new ways to design better studies, and spawn new future research. WROC analysis will help make ROC analysis more general, broadly applicable, practical, accurate, precise, efficient, and user-friendly, and will enable it to make greater impact on evaluation and optimization of medical technologies and medical decision-making.