mlsanityData to 14 Oct 2026

Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shifts

arXiv 2302.06495 ↗First posted 13 Feb 2023cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

Sampling-based methods, e.g., Deep Ensembles and Bayesian Neural Nets have become promising approaches to improve the quality of uncertainty estimation and robust generalization. However, they suffer from a large model size and high latency at test-time, which limits the scalability needed for low-resource devices and real-time applications. To resolve these computational issues, we propose Density-Softmax, a sampling-free deterministic framework via combining a density function built on a Lipschitz-constrained feature extractor with the softmax layer. Theoretically, we show that our model is the solution of minimax uncertainty risk and is distance-aware on feature space, thus reducing the over-confidence of the standard softmax under distribution shifts. Empirically, our method enjoys competitive results with state-of-the-art techniques in terms of uncertainty and robustness, while having a lower number of model parameters and a lower latency at test-time.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 60 of its 83 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

60 of 83 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2016Wide Residual Networkscited by 3,070
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  9. 2017Improved Training of Wasserstein GANscited by 2,235
  10. 2016Density estimation using Real NVPcited by 2,042
  11. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  12. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
38 more
  1. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  2. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  3. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  4. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  5. 2017Deep Neural Networks as Gaussian Processescited by 622
  6. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  7. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 600
  8. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  9. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  10. 2015Weight Uncertainty in Neural Networkscited by 554
  11. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  12. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  13. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  14. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 405
  15. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  16. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  17. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  18. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  19. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  20. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 252
  21. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  22. 2017Concrete Dropoutcited by 227
  23. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 200
  24. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  25. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 195
  26. 2020Training independent subnetworks for robust predictioncited by 177
  27. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  28. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 169
  29. 2020Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Countscited by 164
  30. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  31. 2019Ensemble Distribution Distillationcited by 158
  32. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  33. 2019Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustnesscited by 104
  34. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  35. 2021Soft Calibration Objectives for Neural Networkscited by 92
  36. 2019Distribution Calibration for Regressioncited by 89
  37. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 87
  38. 2021Exploiting Domain-Specific Features to Enhance Domain Generalizationcited by 86

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 3.

12 papers.

  1. 2024Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shiftscited by 7
  2. 2024Variance-Aware Linear UCB with Deep Representation for Neural Contextual Banditscited by 3
  3. 2025Disentangling Uncertainties by Learning Compressed Data Representationcited by 2
  4. 2024Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Taskscited by 1
  5. 2024Source-Free Domain-Invariant Performance Predictioncited by 1
  6. 2025Q-Learning with Shift-Aware Upper Confidence Bound in Non-Stationary Reinforcement Learningcited by 1
  7. 2025On the challenges of detecting MCI using EEG in the wildcited by 0
  8. 2025A Simple and Effective Method for Uncertainty Quantification and OOD Detectioncited by 0
  9. 2025Calibrated Uncertainty Sampling for Active Learningcited by 0
  10. 2026VOLTA: The Surprising Ineffectiveness of Auxiliary Losses for Calibrated Deep Learningcited by 0
  11. 2026U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable explorationcited by 0
  12. 2026Courtroom Analogy: New Perspective on Uncertainty-Aware Classificationcited by 0