# Rate-In: Information-Driven Adaptive Dropout Rates for Improved Inference-Time Uncertainty Estimation

arXiv: https://arxiv.org/abs/2412.07169 · HTML: https://mlsanity.com/paper/2412.07169/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [Improving model calibration with accuracy versus uncertainty optimization](https://mlsanity.com/paper/2012.07923/) (2020)
- [Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers](https://mlsanity.com/paper/1805.08206/) (2018)
- [DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks](https://mlsanity.com/paper/1906.04569/) (2019)
- [Masksembles for Uncertainty Estimation](https://mlsanity.com/paper/2012.08334/) (2020)
- [Bayesian Uncertainty Estimation for Batch Normalized Deep Networks](https://mlsanity.com/paper/1802.06455/) (2018)
- [Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference](https://mlsanity.com/paper/1909.13550/) (2019)
- [Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate](https://mlsanity.com/paper/1910.04858/) (2019)
- [Information Flow in Deep Neural Networks](https://mlsanity.com/paper/2202.06749/) (2022)
- [MedMNIST-C: Comprehensive benchmark and improved classifier robustness by simulating realistic image corruptions](https://mlsanity.com/paper/2406.17536/) (2024)
- [Adaptive Estimators Show Information Compression in Deep Neural Networks](https://mlsanity.com/paper/1902.09037/) (2019)
- [Dropout Injection at Test Time for Post Hoc Uncertainty Quantification in Neural Networks](https://mlsanity.com/paper/2302.02924/) (2023)

## Built on by (most cited first, 7 of 7)

- [Spatially-Aware Evaluation of Segmentation Uncertainty](https://mlsanity.com/paper/2506.16589/) (2025)
- [Dual-domain Multi-path Self-supervised Diffusion Model for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2503.18836/) (2025)
- [Out-of-Distribution Detection in Molecular Complexes via Diffusion Models for Irregular Graphs](https://mlsanity.com/paper/2512.18454/) (2025)
- [Confidence-Uncertainty Boundary Calibration for Bayesian Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/2602.11973/) (2026)
- [CUPID: A Plug-in Framework for Joint Aleatoric and Epistemic Uncertainty Estimation with a Single Model](https://mlsanity.com/paper/2603.10745/) (2026)
- [Monte Carlo Stochastic Depth for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/2604.12719/) (2026)
- [General Incomplete Multimodal Learning via Dynamic Quality Perception](https://mlsanity.com/paper/2607.06943/) (2026)

## Abstract (arXiv)

Accurate uncertainty estimation is crucial for deploying neural networks in risk-sensitive applications such as medical diagnosis. Monte Carlo Dropout is a widely used technique for approximating predictive uncertainty by performing stochastic forward passes with dropout during inference. However, using static dropout rates across all layers and inputs can lead to suboptimal uncertainty estimates, as it fails to adapt to the varying characteristics of individual inputs and network layers. Existing approaches optimize dropout rates during training using labeled data, resulting in fixed inference-time parameters that cannot adjust to new data distributions, compromising uncertainty estimates in Monte Carlo simulations. In this paper, we propose Rate-In, an algorithm that dynamically adjusts dropout rates during inference by quantifying the information loss induced by dropout in each layer's feature maps. By treating dropout as controlled noise injection and leveraging information-theoretic principles, Rate-In adapts dropout rates per layer and per input instance without requiring ground truth labels. By quantifying the functional information loss in feature maps, we adaptively tune dropout rates to maintain perceptual quality across diverse medical imaging tasks and architectural configurations. Our extensive empirical study on synthetic data and real-world medical imaging tasks demonstrates that Rate-In improves calibration and sharpens uncertainty estimates compared to fixed or heuristic dropout rates without compromising predictive performance. Rate-In offers a practical, unsupervised, inference-time approach to optimizing dropout for more reliable predictive uncertainty estimation in critical applications.