# Replication study: Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs

arXiv: https://arxiv.org/abs/1803.04337 · HTML: https://mlsanity.com/paper/1803.04337/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2018.

## Built on by (most cited first, 17 of 17)

- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [An Interpretable Multiple-Instance Approach for the Detection of referable Diabetic Retinopathy from Fundus Images](https://mlsanity.com/paper/2103.01702/) (2021)
- [On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy](https://mlsanity.com/paper/2106.13497/) (2021)
- [Backdoor for Debias: Mitigating Model Bias with Backdoor Attack-based Artificial Bias](https://mlsanity.com/paper/2303.01504/) (2023)
- [DRStageNet: Deep Learning for Diabetic Retinopathy Staging from Fundus Images](https://mlsanity.com/paper/2312.14891/) (2023)
- [Fast Privacy-Preserving Text Classification based on Secure Multiparty Computation](https://mlsanity.com/paper/2101.07365/) (2021)
- [A deep learning model for classification of diabetic retinopathy in eye fundus images based on retinal lesion detection](https://mlsanity.com/paper/2110.07745/) (2021)
- [A systematic review of transfer learning based approaches for diabetic retinopathy detection](https://mlsanity.com/paper/2105.13793/) (2021)
- [Debiasing Backdoor Attack: A Benign Application of Backdoor Attack in Eliminating Data Bias](https://mlsanity.com/paper/2202.10582/) (2022)
- [Deep Semi-Supervised and Self-Supervised Learning for Diabetic Retinopathy Detection](https://mlsanity.com/paper/2208.02408/) (2022)
- [A ResNet is All You Need? Modeling A Strong Baseline for Detecting Referable Diabetic Retinopathy in Fundus Images](https://mlsanity.com/paper/2210.03180/) (2022)
- [AC-Norm: Effective Tuning for Medical Image Analysis via Affine Collaborative Normalization](https://mlsanity.com/paper/2307.15282/) (2023)
- [Dive into Self-Supervised Learning for Medical Image Analysis: Data, Models and Tasks](https://mlsanity.com/paper/2209.12157/) (2022)
- [Data Set Terminology of Deep Learning in Medicine: A Historical Review and Recommendation](https://mlsanity.com/paper/2404.19303/) (2024)
- [Explainable Deep Learning for Cataract Detection in Retinal Images: A Dual-Eye and Knowledge Distillation Approach](https://mlsanity.com/paper/2509.22696/) (2025)
- [Rethinking Benchmarks for Differentially Private Image Classification](https://mlsanity.com/paper/2601.17189/) (2026)
- [Dual-Resolution Attention-Gated Deep Learning with Ordinal Regression for Diabetic Retinopathy Grading: A Quantified Assessment of Cross-Domain Generalization](https://mlsanity.com/paper/2604.17341/) (2026)

## Abstract (arXiv)

Replication studies are essential for validation of new methods, and are crucial to maintain the high standards of scientific publications, and to use the results in practice. We have attempted to replicate the main method in 'Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs' published in JAMA 2016; 316(22). We re-implemented the method since the source code is not available, and we used publicly available data sets. The original study used non-public fundus images from EyePACS and three hospitals in India for training. We used a different EyePACS data set from Kaggle. The original study used the benchmark data set Messidor-2 to evaluate the algorithm's performance. We used the same data set. In the original study, ophthalmologists re-graded all images for diabetic retinopathy, macular edema, and image gradability. There was one diabetic retinopathy grade per image for our data sets, and we assessed image gradability ourselves. Hyper-parameter settings were not described in the original study. But some of these were later published. We were not able to replicate the original study. Our algorithm's area under the receiver operating curve (AUC) of 0.94 on the Kaggle EyePACS test set and 0.80 on Messidor-2 did not come close to the reported AUC of 0.99 in the original study. This may be caused by the use of a single grade per image, different data, or different not described hyper-parameter settings. This study shows the challenges of replicating deep learning, and the need for more replication studies to validate deep learning methods, especially for medical image analysis. Our source code and instructions are available at: https://github.com/mikevoets/jama16-retina-replication