mlsanityData to 7 Oct 2026

Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem

arXiv 1812.05720 ↗First posted 13 Dec 2018cs.LG cs.CV stat.MLcited by 323

Not on the map. Why?

Abstract (arXiv)

Classifiers used in the wild, in particular for safety-critical systems, should not only have good generalization properties but also should know when they don't know, in particular make low confidence predictions far away from the training data. We show that ReLU type neural networks which yield a piecewise linear classifier function fail in this regard as they produce almost always high confidence predictions far away from the training data. For bounded domains like images we propose a new robust optimization technique similar to adversarial training which enforces low confidence predictions far away from the training data. We show that this technique is surprisingly effective in reducing the confidence of predictions far away from the training data while maintaining high confidence predictions and test error on the original classification task compared to standard training.

Key facts. Cited by 323 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 75; 2022: 82; 2023: 68; 2024: 49; 2025: 33; 2026: 16.

323 papers.

Sort:

By area:

  1. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  2. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  3. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  4. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  5. 2021Dynamic Neural Networks: A Surveycited by 405
  6. 2022VOS: Learning What You Don't Know by Virtual Outlier Synthesiscited by 300
  7. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
  8. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  9. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  10. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  11. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166
  12. 2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
38 more
  1. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  2. 2021SegmentMeIfYouCan: A Benchmark for Anomaly Segmentationcited by 150
  3. 2023In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluationcited by 132
  4. 2023Non-Parametric Outlier Synthesiscited by 129
  5. 2021Learning Placeholders for Open-Set Recognitioncited by 121
  6. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  7. 2021Influence Selection for Active Learningcited by 82
  8. 2021Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classificationcited by 79
  9. 2021Provable Guarantees for Understanding Out-of-distribution Detectioncited by 78
  10. 2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
  11. 2021Explainable Artificial Intelligence (XAI) on TimeSeries Data: A Surveycited by 71
  12. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  13. 2022Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wildcited by 70
  14. 2022RbA: Segmenting Unknown Regions Rejected by Allcited by 67
  15. 2023Energy-based Out-of-Distribution Detection for Graph Neural Networkscited by 67
  16. 2023Nearest Neighbor Guidance for Out-of-Distribution Detectioncited by 64
  17. 2021Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularizationcited by 62
  18. 2021Understanding Softmax Confidence and Uncertaintycited by 62
  19. 2022Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detectioncited by 59
  20. 2021Open-set Label Noise Can Improve Robustness Against Inherent Label Noisecited by 58
  21. 2022Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attackscited by 58
  22. 2022Confidence Score for Source-Free Unsupervised Domain Adaptationcited by 58
  23. 2021DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modelingcited by 55
  24. 2022Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Seriescited by 54
  25. 2021DOCTOR: A Simple Method for Detecting Misclassification Errorscited by 52
  26. 2022Semi-Supervised and Unsupervised Deep Visual Learning: A Surveycited by 52
  27. 2022A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibrationcited by 49
  28. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  29. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  30. 2022Boosting Out-of-distribution Detection with Typical Featurescited by 48
  31. 2021Energy-Based Open-World Uncertainty Modeling for Confidence Calibrationcited by 46
  32. 2022Uncertainty-guided Source-free Domain Adaptationcited by 46
  33. 2021Posterior Meta-Replay for Continual Learningcited by 45
  34. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  35. 2022Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognitioncited by 44
  36. 2022Harnessing Out-Of-Distribution Examples via Augmenting Content and Stylecited by 43
  37. 2021Neural Mean Discrepancy for Efficient Out-of-Distribution Detectioncited by 42
  38. 2023ID-like Prompt Learning for Few-Shot Out-of-Distribution Detectioncited by 42

50 most cited of 323 All 323 (JSON)