mlsanityData to 7 Oct 2026

A Closer Look at AUROC and AUPRC under Class Imbalance

arXiv 2401.06091 ↗First posted 11 Jan 2024cs.LG stat.MEcited by 56

Not on the map. Why?

Abstract (arXiv)

In machine learning (ML), a widespread claim is that the area under the precision-recall curve (AUPRC) is a superior metric for model comparison to the area under the receiver operating characteristic (AUROC) for tasks with class imbalance. This paper refutes this notion on two fronts. First, we theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, establishing clearly that AUPRC is not generally superior in cases of class imbalance. We further show that AUPRC can be a harmful metric as it can unduly favor model improvements in subpopulations with more frequent positive labels, heightening algorithmic disparities. Next, we empirically support our theory using experiments on both semi-synthetic and real-world fairness datasets. Prompted by these insights, we conduct a review of over 1.5 million scientific papers to understand the origin of this invalid claim, finding that it is often made without citation, misattributed to papers that do not argue this point, and aggressively over-generalized from source arguments. Our findings represent a dual contribution: a significant technical advancement in understanding the relationship between AUROC and AUPRC and a stark warning about unchecked assumptions in the ML community.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 101 of its 169 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

101 of 169 references.

References · 101

  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  2. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  3. 2021DRAEM -- A discriminatively trained reconstruction embedding for surface anomaly detectioncited by 510
  4. 2020A Unifying Review of Deep and Shallow Anomaly Detectioncited by 423
  5. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 397
  6. 2017Multitask learning and benchmarking with clinical time series datacited by 287
  7. 2020Fairness without Demographics through Adversarially Reweighted Learningcited by 251
  8. 2020Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervisioncited by 235
  9. 2020Foundations and modelling of dynamic networks using Dynamic Graph Neural Networks: A surveycited by 123
  10. 2019Interpolation-Prediction Networks for Irregularly Sampled Time Seriescited by 89
  11. 2018MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcarecited by 72
  12. 2015Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptionscited by 64
38 more
  1. 2019Deep Weakly-supervised Anomaly Detectioncited by 59
  2. 2017Deep Over-sampling Framework for Classifying Imbalanced Datacited by 58
  3. 2019ConCare: Personalized Clinical Feature Embedding via Capturing the Healthcare Contextcited by 56
  4. 2020Differentially Private Synthetic Medical Data Generation using Convolutional GANscited by 52
  5. 2019Detecting semantic anomaliescited by 50
  6. 2018An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videoscited by 45
  7. 2018Automated software vulnerability detection with machine learningcited by 34
  8. 2015Evaluating Link Prediction Methodscited by 31
  9. 2019Toward Interpretable Music Tagging with Self-Attentioncited by 29
  10. 2020Molecular machine learning with conformer ensemblescited by 27
  11. 2020Benchmarking Unsupervised Outlier Detection with Realistic Synthetic Datacited by 25
  12. 2019Multimodal Machine Learning-based Knee Osteoarthritis Progression Prediction from Plain Radiographs and Clinical Datacited by 23
  13. 2018Asymmetric Loss Functions and Deep Densely Connected Networks for Highly Imbalanced Medical Image Segmentation: Application to Multiple Sclerosis Lesion Detectioncited by 22
  14. 2019RAPID: Early Classification of Explosive Transients using Deep Learningcited by 17
  15. 2022Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcarecited by 16
  16. 2020Characterizing the Value of Information in Medical Notescited by 12
  17. 2021Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoringcited by 11
  18. 2022A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithmscited by 10
  19. 2019Unsupervised Boosting-based Autoencoder Ensembles for Outlier Detectioncited by 8
  20. 2020Precision-Recall Curve (PRC) Classification Treescited by 8
  21. 2021Deep Learning-based Damage Mapping with InSAR Coherence Time Seriescited by 8
  22. 2022Towards a consistent interpretation of AIOps modelscited by 8
  23. 2020An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning applied to Gastrointestinal Tract Abnormality Classificationcited by 7
  24. 2020AttentionDDI: Siamese Attention-based Deep Learning method for drug-drug interaction predictionscited by 7
  25. 2021Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognitioncited by 7
  26. 2016What can we Learn from Predictive Modeling?cited by 6
  27. 2018The Effectiveness of Multitask Learning for Phenotyping with Electronic Health Records Datacited by 6
  28. 2020A multimodal approach for multi-label movie genre classificationcited by 6
  29. 2021Time-Based CAN Intrusion Detection Benchmarkcited by 6
  30. 2021Key Technology Considerations in Developing and Deploying Machine Learning Models in Clinical Radiology Practicecited by 6
  31. 2021Three-level Hierarchical Transformer Networks for Long-sequence and Multiple Clinical Documents Classificationcited by 6
  32. 2021Deep AUC Maximization for Medical Image Classification: Challenges and Opportunitiescited by 6
  33. 2022Classic Graph Structural Features Outperform Factorization-Based Graph Embedding Methods on Community Labelingcited by 6
  34. 2023Structural Attention-Based Recurrent Variational Autoencoder for Highway Vehicle Anomaly Detectioncited by 6
  35. 2019A comparative evaluation of novelty detection algorithms for discrete sequencescited by 5
  36. 2021Abusive Language Detection in Heterogeneous Contexts: Dataset Collection and the Role of Supervised Attentioncited by 5
  37. 2022A Meta-level Analysis of Online Anomaly Detectorscited by 5
  38. 2023Towards Reliable Assessments of Demographic Disparities in Multi-Label Image Classifierscited by 5

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 23; 2026: 23.

56 papers.

Sort:

By area:

  1. 2024Developing ChemDFM as a large language foundation model for chemistrycited by 51
  2. 2024ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaborationcited by 23
  3. 2024REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Modelscited by 21
  4. 2024EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generationcited by 16
  5. 2024Enhancing clinical decision support with physiological waveforms -- a multimodal benchmark in emergency carecited by 13
  6. 2024Training a high-performance retinal foundation model with half-the-data and 400 times less computecited by 9
  7. 2024TimEHR: Image-based Time Series Generation for Electronic Health Recordscited by 8
  8. 2025A Collection of Innovations in Medical AI for patient records in 2024cited by 7
  9. 2025Benchmarking ECG FMs: A Reality Check Across Clinical Taskscited by 6
  10. 2026Not All Inputs Are Valid: Towards Open-Set Video Moment Retrieval Using Languagecited by 6
  11. 2024Abnormality Prediction and Forecasting of Laboratory Values from Electrocardiogram Signals Using Multimodal Deep Learningcited by 5
  12. 2025Learning Contrastive Multimodal Fusion with Improved Modality Dropout for Disease Detection and Predictioncited by 5
38 more
  1. 2024Explaining Time Series Classification Predictions via Causal Attributionscited by 4
  2. 2025Cohort-Based Active Modality Acquisitioncited by 4
  3. 2025We Need to Rethink Benchmarking in Anomaly Detectioncited by 4
  4. 2025Synthetic Data Generation for Augmenting Small Samplescited by 3
  5. 2025Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costscited by 3
  6. 2025DACTYL: Diverse Adversarial Corpus of Texts Yielded from Large Language Modelscited by 3
  7. 2025Rethinking Tokenization for Clinical Time Series: When Less is Morecited by 3
  8. 2025Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scalescited by 2
  9. 2025Critical Appraisal of Fairness Metrics in Clinical Predictive AIcited by 2
  10. 2025Limitations of Public Chest Radiography Datasets for Artificial Intelligence: Label Quality, Domain Shift, Bias and Evaluation Challengescited by 2
  11. 2025ReTabAD: A Benchmark for Restoring Semantic Context in Tabular Anomaly Detectioncited by 2
  12. 2025Traj-CoA: Patient Trajectory Modeling via Chain-of-Agents for Lung Cancer Risk Predictioncited by 2
  13. 2026A multi-agent system for spine MRI report generation from multi-sequence imagingcited by 2
  14. 2025Med-gte-hybrid: A contextual embedding transformer model for extracting actionable information from clinical textscited by 1
  15. 2025The Role of Model Confidence on Bias Effects in Measured Uncertainties for Vision-Language Modelscited by 1
  16. 2025Unsupervised Deep Generative Models for Anomaly Detection in Neuroimaging: A Systematic Scoping Reviewcited by 1
  17. 2025KAT-GNN: A Knowledge-Augmented Temporal Graph Neural Network for Risk Prediction in Electronic Health Recordscited by 1
  18. 2026A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarkingcited by 1
  19. 2026LabelKAN -- Kolmogorov-Arnold Networks for Inter-Label Learning: Avian Community Learningcited by 1
  20. 2026Trajectory-Aware Reliability Modeling of Democratic Systemscited by 1
  21. 2026Does Your Wildfire Prediction Model Actually Work, or Just Score Well?cited by 1
  22. 2026Auditing Frame-Level AUC in Weakly Supervised Video Anomaly Detection: Granularity, Resolution, and Scene Biascited by 1
  23. 2026When Deep Research Agents Stagnate: Enhancing Reasoning with Retrieval-Aware Agent Controlcited by 1
  24. 2024ALPEC: A Comprehensive Evaluation Framework and Dataset for Machine Learning-Based Arousal Detection in Clinical Practicecited by 0
  25. 2025Predicting Chest Radiograph Findings from Electrocardiograms Using Interpretable Machine Learningcited by 0
  26. 2025Fourier Transform Multiple Instance Learning for Whole Slide Image Classificationcited by 0
  27. 2025Needles in the Landscape: Semi-Supervised Pseudolabeling for Archaeological Site Discovery under Label Scarcitycited by 0
  28. 2025Partial VOROS: A Cost-aware Performance Metric for Binary Classifiers with Precision and Capacity Constraintscited by 0
  29. 2025A Machine Learning Approach for Detection of Mental Health Conditions and Cyberbullying from Social Mediacited by 0
  30. 2026Anomaly Detection via Mean Shift Density Enhancementcited by 0
  31. 2026CoE: Collaborative Entropy for Uncertainty Quantification in Agentic Multi-LLM Systemscited by 0
  32. 2026Towards Explainable Stakeholder-Aware Requirements Prioritisation in Aged-Care Digital Healthcited by 0
  33. 2026Development and evaluation of CADe systems in low-prevalence setting: The RARE25 challenge for early detection of Barrett's neoplasiacited by 0
  34. 2026Kurtosis-Guided Denoising Score Matching for Tabular Anomaly Detectioncited by 0
  35. 2026Decomposing the Generalization Gap in PROTAC Activity Prediction: Variance Attribution and the Inter-Laboratory Ceilingcited by 0
  36. 2026Stream Assembly Is an Uncontrolled Treatment in Streaming Intrusion-Detection Benchmarkscited by 0
  37. 2026Not All NVFP4 QAT Recipes Are Equal: How Architecture and Scale Shape Model Quality for Anomaly Segmentationcited by 0
  38. 2026Beyond Accuracy: Measuring Logical Compliance of Predictive Modelscited by 0

50 most cited of 56 All 56 (JSON)