# A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks

arXiv: https://arxiv.org/abs/1912.10481 · HTML: https://mlsanity.com/paper/1912.10481/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2019.

## Built on by (most cited first, 30 of 62)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [Benchmarking Simulation-Based Inference](https://mlsanity.com/paper/2101.04653/) (2021)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [Revisiting Design Choices in Offline Model-Based Reinforcement Learning](https://mlsanity.com/paper/2110.04135/) (2021)
- [Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks](https://mlsanity.com/paper/2211.12717/) (2022)
- [On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification](https://mlsanity.com/paper/2203.16481/) (2022)
- [Tractable Function-Space Variational Inference in Bayesian Neural Networks](https://mlsanity.com/paper/2312.17199/) (2023)
- [Dangers of Bayesian Model Averaging under Covariate Shift](https://mlsanity.com/paper/2106.11905/) (2021)
- [Quantification of Uncertainty with Adversarial Models](https://mlsanity.com/paper/2307.03217/) (2023)
- [Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty](https://mlsanity.com/paper/2311.08309/) (2023)
- [Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning](https://mlsanity.com/paper/2111.03577/) (2021)
- [On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study](https://mlsanity.com/paper/2304.08653/) (2023)
- [Shifts 2.0: Extending The Dataset of Real Distributional Shifts](https://mlsanity.com/paper/2206.15407/) (2022)
- [Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift](https://mlsanity.com/paper/2306.12306/) (2023)
- [Natural Posterior Network: Deep Bayesian Uncertainty for Exponential Family Distributions](https://mlsanity.com/paper/2105.04471/) (2021)
- [Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue](https://mlsanity.com/paper/2105.10155/) (2021)
- [Composite Bayesian Optimization In Function Spaces Using NEON -- Neural Epistemic Operator Networks](https://mlsanity.com/paper/2404.03099/) (2024)
- [Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference](https://mlsanity.com/paper/2305.14765/) (2023)
- [Posterior Uncertainty Quantification in Neural Networks using Data Augmentation](https://mlsanity.com/paper/2403.12729/) (2024)
- [Deep Classifiers with Label Noise Modeling and Distance Awareness](https://mlsanity.com/paper/2110.02609/) (2021)
- [Uncertainty-aware deep learning methods for robust diabetic retinopathy classification](https://mlsanity.com/paper/2201.09042/) (2022)
- [Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitions](https://mlsanity.com/paper/2401.04305/) (2024)
- [Training Bayesian Neural Networks with Sparse Subspace Variational Inference](https://mlsanity.com/paper/2402.11025/) (2024)
- [Benchmarking CNN on 3D Anatomical Brain MRI: Architectures, Data Augmentation and Deep Ensemble Learning](https://mlsanity.com/paper/2106.01132/) (2021)
- [PCCT: Progressive Class-Center Triplet Loss for Imbalanced Medical Image Classification](https://mlsanity.com/paper/2207.04793/) (2022)

## Abstract (arXiv)

Evaluation of Bayesian deep learning (BDL) methods is challenging. We often seek to evaluate the methods' robustness and scalability, assessing whether new tools give `better' uncertainty estimates than old ones. These evaluations are paramount for practitioners when choosing BDL tools on-top of which they build their applications. Current popular evaluations of BDL methods, such as the UCI experiments, are lacking: Methods that excel with these experiments often fail when used in application such as medical or automotive, suggesting a pertinent need for new benchmarks in the field. We propose a new BDL benchmark with a diverse set of tasks, inspired by a real-world medical imaging application on diabetic retinopathy diagnosis. Visual inputs (512x512 RGB images of retinas) are considered, where model uncertainty is used for medical pre-screening---i.e. to refer patients to an expert when model diagnosis is uncertain. Methods are then ranked according to metrics derived from expert-domain to reflect real-world use of model uncertainty in automated diagnosis. We develop multiple tasks that fall under this application, including out-of-distribution detection and robustness to distribution shift. We then perform a systematic comparison of well-tuned BDL techniques on the various tasks. From our comparison we conclude that some current techniques which solve benchmarks such as UCI `overfit' their uncertainty to the dataset---when evaluated on our benchmark these underperform in comparison to simpler baselines. The code for the benchmark, its baselines, and a simple API for evaluating new BDL tools are made available at https://github.com/oatml/bdl-benchmarks.