# Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy

arXiv: https://arxiv.org/abs/1710.01711 · HTML: https://mlsanity.com/paper/1710.01711/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2017.

## Built on by (most cited first, 30 of 44)

- [Explaining in Style: Training a GAN to explain a classifier in StyleSpace](https://mlsanity.com/paper/2104.13369/) (2021)
- [Guided Integrated Gradients: An Adaptive Path Method for Removing Noise](https://mlsanity.com/paper/2106.09788/) (2021)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond](https://mlsanity.com/paper/2102.01998/) (2021)
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [Applications of Deep Learning in Fundus Images: A Review](https://mlsanity.com/paper/2101.09864/) (2021)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [Balanced-MixUp for Highly Imbalanced Medical Image Classification](https://mlsanity.com/paper/2109.09850/) (2021)
- [Active label cleaning for improved dataset quality under resource constraints](https://mlsanity.com/paper/2109.00574/) (2021)
- [Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)](https://mlsanity.com/paper/2102.01564/) (2021)
- [MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertise](https://mlsanity.com/paper/2405.11793/) (2024)
- [Learning Robust Representation for Joint Grading of Ophthalmic Diseases via Adaptive Curriculum and Feature Disentanglement](https://mlsanity.com/paper/2207.04183/) (2022)
- [Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization](https://mlsanity.com/paper/2207.13676/) (2022)
- [Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making](https://mlsanity.com/paper/2308.04375/) (2023)
- [Explainable Autonomous Robots: A Survey and Perspective](https://mlsanity.com/paper/2105.02658/) (2021)
- [Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders](https://mlsanity.com/paper/2212.07050/) (2022)
- [SSiT: Saliency-guided Self-supervised Image Transformer for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2210.10969/) (2022)
- [Training a high-performance retinal foundation model with half-the-data and 400 times less compute](https://mlsanity.com/paper/2405.00117/) (2024)
- [Uncertainty-aware deep learning methods for robust diabetic retinopathy classification](https://mlsanity.com/paper/2201.09042/) (2022)
- [Dual Branch Deep Learning Network for Detection and Stage Grading of Diabetic Retinopathy](https://mlsanity.com/paper/2308.09945/) (2023)
- [An Interpretable Multiple-Instance Approach for the Detection of referable Diabetic Retinopathy from Fundus Images](https://mlsanity.com/paper/2103.01702/) (2021)
- [DRStageNet: Deep Learning for Diabetic Retinopathy Staging from Fundus Images](https://mlsanity.com/paper/2312.14891/) (2023)
- [A Framework of Explanation Generation toward Reliable Autonomous Robots](https://mlsanity.com/paper/2105.02670/) (2021)
- [Detecting Severity of Diabetic Retinopathy from Fundus Images: A Transformer Network-based Review](https://mlsanity.com/paper/2301.00973/) (2023)
- [Learning from few examples: Classifying sex from retinal images via deep learning](https://mlsanity.com/paper/2207.09624/) (2022)
- [Deep Semi-Supervised and Self-Supervised Learning for Diabetic Retinopathy Detection](https://mlsanity.com/paper/2208.02408/) (2022)
- [Towards Assessing Data Bias in Clinical Trials](https://mlsanity.com/paper/2212.09633/) (2022)
- [VR-FuseNet: A Fusion of Heterogeneous Fundus Data and Explainable Deep Network for Diabetic Retinopathy Classification](https://mlsanity.com/paper/2504.21464/) (2025)
- [OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment](https://mlsanity.com/paper/2607.25545/) (2026)
- [Information-theoretic Classification Accuracy: A Criterion that Guides Data-driven Combination of Ambiguous Outcome Labels in Multi-class Classification](https://mlsanity.com/paper/2109.00582/) (2021)

## Abstract (arXiv)

Diabetic retinopathy (DR) and diabetic macular edema are common complications of diabetes which can lead to vision loss. The grading of DR is a fairly complex process that requires the detection of fine features such as microaneurysms, intraretinal hemorrhages, and intraretinal microvascular abnormalities. Because of this, there can be a fair amount of grader variability. There are different methods of obtaining the reference standard and resolving disagreements between graders, and while it is usually accepted that adjudication until full consensus will yield the best reference standard, the difference between various methods of resolving disagreements has not been examined extensively. In this study, we examine the variability in different methods of grading, definitions of reference standards, and their effects on building deep learning models for the detection of diabetic eye disease. We find that a small set of adjudicated DR grades allows substantial improvements in algorithm performance. The resulting algorithm's performance was on par with that of individual U.S. board-certified ophthalmologists and retinal specialists.