# Atomic Calibration of LLMs in Long-Form Generations

arXiv: https://arxiv.org/abs/2410.13246 · HTML: https://mlsanity.com/paper/2410.13246/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification](https://mlsanity.com/paper/2403.04696/) (2024)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus](https://mlsanity.com/paper/2311.13230/) (2023)
- [LUQ: Long-text Uncertainty Quantification for LLMs](https://mlsanity.com/paper/2403.20279/) (2024)
- [On the Calibration of Large Language Models and Alignment](https://mlsanity.com/paper/2311.13240/) (2023)
- [Calibrating Large Language Models Using Their Generations Only](https://mlsanity.com/paper/2403.05973/) (2024)
- [Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators](https://mlsanity.com/paper/2406.13415/) (2024)
- [Linguistic Calibration of Long-Form Generations](https://mlsanity.com/paper/2404.00474/) (2024)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses](https://mlsanity.com/paper/2310.19208/) (2023)
- [Combining Confidence Elicitation and Sample-based Methods for Uncertainty Quantification in Misinformation Mitigation](https://mlsanity.com/paper/2401.08694/) (2024)
- [Uncertainty-aware Language Modeling for Selective Question Answering](https://mlsanity.com/paper/2311.15451/) (2023)
- [Do We Need Language-Specific Fact-Checking Models? The Case of Chinese](https://mlsanity.com/paper/2401.15498/) (2024)
- [Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations](https://mlsanity.com/paper/2402.05629/) (2024)
- [Atomic Self-Consistency for Better Long Form Generations](https://mlsanity.com/paper/2405.13131/) (2024)

## Built on by (most cited first, 24 of 24)

- [A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs](https://mlsanity.com/paper/2505.08200/) (2025)
- [LoGU: Long-form Generation with Uncertainty Expressions](https://mlsanity.com/paper/2410.14309/) (2024)
- [Demystifying Multi-Agent Debate: The Role of Confidence and Diversity](https://mlsanity.com/paper/2601.19921/) (2026)
- [LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generations](https://mlsanity.com/paper/2505.23912/) (2025)
- [Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing](https://mlsanity.com/paper/2507.10587/) (2025)
- [Confidence Estimation for LLMs in Multi-turn Interactions](https://mlsanity.com/paper/2601.02179/) (2026)
- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generation](https://mlsanity.com/paper/2505.16922/) (2025)
- [Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models](https://mlsanity.com/paper/2510.03136/) (2025)
- [Are LLM Decisions Faithful to Verbal Confidence?](https://mlsanity.com/paper/2601.07767/) (2026)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study](https://mlsanity.com/paper/2602.17431/) (2026)
- [Code Is More Than Text: Uncertainty Estimation for Code Generation](https://mlsanity.com/paper/2606.09577/) (2026)
- [GrACE: A Generative Approach to Better Confidence Elicitation and Efficient Test-Time Scaling in Large Language Models](https://mlsanity.com/paper/2509.09438/) (2025)
- [All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning](https://mlsanity.com/paper/2509.12908/) (2025)
- [Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification](https://mlsanity.com/paper/2605.28500/) (2026)
- [When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs](https://mlsanity.com/paper/2606.30814/) (2026)
- [Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Models](https://mlsanity.com/paper/2605.31220/) (2026)
- [Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents](https://mlsanity.com/paper/2608.11552/) (2026)
- [Knowing Your Uncertainty -- On the application of LLM in social sciences](https://mlsanity.com/paper/2512.05461/) (2025)
- [Task-Aware Calibration: Provably Optimal Decoding in LLMs](https://mlsanity.com/paper/2605.10202/) (2026)
- [Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth](https://mlsanity.com/paper/2607.03870/) (2026)
- [When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study](https://mlsanity.com/paper/2608.24492/) (2026)
- [Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models](https://mlsanity.com/paper/2609.22206/) (2026)

## Abstract (arXiv)

Large language models (LLMs) often suffer from hallucinations, posing significant challenges for real-world applications. Confidence calibration, as an effective indicator of hallucination, is thus essential to enhance the trustworthiness of LLMs. Prior work mainly focuses on short-form tasks using a single response-level score (macro calibration), which is insufficient for long-form outputs that may contain both accurate and inaccurate claims. In this work, we systematically study atomic calibration, which evaluates factuality calibration at a fine-grained level by decomposing long responses into atomic claims. We further categorize existing confidence elicitation methods into discriminative and generative types, and propose two new confidence fusion strategies to improve calibration. Our experiments demonstrate that LLMs exhibit poorer calibration at the atomic level during long-form generation. More importantly, atomic calibration uncovers insightful patterns regarding the alignment of confidence methods and the changes of confidence throughout generation. This sheds light on future research directions for confidence estimation in long-form generation.