Pretraining with random noise for uncertainty calibration
Not on the map. Why?
Abstract (arXiv)
Uncertainty calibration is crucial for various machine learning applications, yet it remains challenging. Many models exhibit hallucinations - confident yet inaccurate responses - due to miscalibrated confidence. Here, we show that the common practice of random initialization in deep learning, often considered a standard technique, is an underlying cause of this miscalibration, leading to excessively high confidence in untrained networks. Our method, inspired by developmental neuroscience, addresses this issue by simply pretraining networks with random noise and labels, reducing overconfidence and bringing initial confidence levels closer to chance. This ensures optimal calibration, aligning confidence with accuracy during subsequent data training, without the need for additional pre- or post-processing. Pre-calibrated networks excel at identifying "unknown data," showing low confidence for out-of-distribution inputs, thereby resolving confidence miscalibration.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 82 references that are in the map. First posted on arXiv 23 Dec 2024.
Builds on
33 of 82 references.
References · 33
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2016Densely Connected Convolutional Networkscited by 6,720
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
- 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
- 2020Energy-based Out-of-distribution Detectioncited by 1,299
21 more
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,186
- 2018Deep Anomaly Detection with Outlier Exposurecited by 942
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2018Object Hallucination in Image Captioningcited by 669
- 2018Do Deep Generative Models Know What They Don't Know?cited by 443
- 2017Deep Learning for Medical Image Analysiscited by 401
- 2021Revisiting the Calibration of Modern Neural Networkscited by 390
- 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
- 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
- 2017Noisy Networks for Explorationcited by 266
- 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
- 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
- 2019Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentationcited by 163
- 2019Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Controlcited by 61
- 2024Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Modelscited by 54
- 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
- 2024On Overcoming Miscalibrated Conversational Priors in LLM-based Chatbotscited by 24
- 2019Model-based Reinforcement Learning for Predictions and Control for Limit Order Bookscited by 14
- 2022Confidence Calibration for Object Detection and Segmentationcited by 5
- 2024Pretraining with Random Noise for Fast and Robust Learning without Weight Transportcited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.