# UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generation

arXiv: https://arxiv.org/abs/2505.16922 · HTML: https://mlsanity.com/paper/2505.16922/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales](https://mlsanity.com/paper/2405.20974/) (2024)
- [Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models](https://mlsanity.com/paper/2302.13439/) (2023)
- [Teaching Large Language Models to Express Knowledge Boundary from Their Own Signals](https://mlsanity.com/paper/2406.10881/) (2024)
- [WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries](https://mlsanity.com/paper/2407.17468/) (2024)
- [Atomic Calibration of LLMs in Long-Form Generations](https://mlsanity.com/paper/2410.13246/) (2024)
- [Enhancing Confidence Expression in Large Language Models Through Learning from Past Experience](https://mlsanity.com/paper/2404.10315/) (2024)
- [LoGU: Long-form Generation with Uncertainty Expressions](https://mlsanity.com/paper/2410.14309/) (2024)
- [LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation](https://mlsanity.com/paper/2309.12455/) (2023)
- [Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuning](https://mlsanity.com/paper/2406.10099/) (2024)
- [Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations](https://mlsanity.com/paper/2402.05629/) (2024)
- [Multi-group Uncertainty Quantification for Long-form Text Generation](https://mlsanity.com/paper/2407.21057/) (2024)
- [Graph-based Uncertainty Metrics for Long-form Language Model Outputs](https://mlsanity.com/paper/2410.20783/) (2024)

## Built on by (most cited first, 11 of 11)

- [Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents](https://mlsanity.com/paper/2505.22655/) (2025)
- [Demystifying Multi-Agent Debate: The Role of Confidence and Diversity](https://mlsanity.com/paper/2601.19921/) (2026)
- [LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generations](https://mlsanity.com/paper/2505.23912/) (2025)
- [Confidence Estimation for LLMs in Multi-turn Interactions](https://mlsanity.com/paper/2601.02179/) (2026)
- [Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models](https://mlsanity.com/paper/2510.03136/) (2025)
- [Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text](https://mlsanity.com/paper/2606.18471/) (2026)
- [GrACE: A Generative Approach to Better Confidence Elicitation and Efficient Test-Time Scaling in Large Language Models](https://mlsanity.com/paper/2509.09438/) (2025)
- [All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning](https://mlsanity.com/paper/2509.12908/) (2025)
- [When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs](https://mlsanity.com/paper/2606.30814/) (2026)
- [UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning](https://mlsanity.com/paper/2604.05517/) (2026)
- [IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking](https://mlsanity.com/paper/2606.09709/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are prone to hallucination, particularly in long-form generations. A promising direction to mitigate hallucination is to teach LLMs to express uncertainty explicitly when they lack sufficient knowledge. However, existing work lacks direct and fair evaluation of LLMs' ability to express uncertainty effectively in long-form generation. To address this gap, we first introduce UNCLE, a benchmark designed to evaluate uncertainty expression in both long- and short-form question answering (QA). UNCLE covers five domains and includes more than 1,000 entities, each with paired short- and long-form QA items. Our dataset is the first to directly link short- and long-form QA through aligned questions and gold-standard answers. Along with UNCLE, we propose a suite of new metrics to assess the models' capabilities to selectively express uncertainty. We then demonstrate that current models fail to convey uncertainty appropriately in long-form generation. We further explore both prompt-based and training-based methods to improve models' performance, with the training-based methods yielding greater gains. Further analysis of alignment gaps between short- and long-form uncertainty expression highlights promising directions for future research using UNCLE.