# Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection

arXiv: https://arxiv.org/abs/2608.17965 · HTML: https://mlsanity.com/paper/2608.17965/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 18 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://mlsanity.com/paper/2008.06448/) (2020)
- [Log-based Anomaly Detection Without Log Parsing](https://mlsanity.com/paper/2108.01955/) (2021)
- [Log-based Anomaly Detection with Deep Learning: How Far Are We?](https://mlsanity.com/paper/2202.04301/) (2022)
- [Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs](https://mlsanity.com/paper/2008.09340/) (2020)
- [On Calibrating Semantic Segmentation Models: Analyses and An Algorithm](https://mlsanity.com/paper/2212.12053/) (2022)
- [LogLLM: Log-based Anomaly Detection Using Large Language Models](https://mlsanity.com/paper/2411.08561/) (2024)
- [A Critical Review of Common Log Data Sets Used for Evaluation of Sequence-based Anomaly Detection Techniques](https://mlsanity.com/paper/2309.02854/) (2023)
- [Towards Reliable Rare Category Analysis on Graphs via Individual Calibration](https://mlsanity.com/paper/2307.09858/) (2023)
- [Adapting Large Language Models for Parameter-Efficient Log Anomaly Detection](https://mlsanity.com/paper/2503.08045/) (2025)

## Abstract (arXiv)

Online log anomaly detection is critical for maintaining the reliability of large-scale computing systems. Although recent language model-based log anomaly detectors achieve strong detection performance, their confidence estimates remain poorly calibrated. We show that these detectors frequently assign excessive confidence to incorrect predictions, particularly for anomalous logs under severe class imbalance. Moreover, confidence on erroneous predictions remains persistently high even when conventional calibration metrics indicate good calibration, creating a critical reliability gap for operational monitoring systems. To address this issue, we propose Log Reconstruction and Distance (LoRD), a lightweight post-hoc calibration framework for reliable log anomaly detection. LoRD learns prediction-route-specific reliability models from latent representations of correctly classified validation samples and estimates prediction reliability through route-wise reconstruction distances. Based on the estimated reliability, LoRD selectively recalibrates high-risk predictions to suppress overconfident errors while preserving reliable predictions. Extensive experiments on four large-scale log benchmark datasets and multiple language model-based detectors demonstrate that LoRD consistently improves confidence reliability and substantially reduces overconfident anomaly-related errors without sacrificing anomaly detection performance.