# Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoning

arXiv: https://arxiv.org/abs/2509.02401 · HTML: https://mlsanity.com/paper/2509.02401/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 44 of its 69 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 30 of 44)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.01441/) (2025)
- [FineSurE: Fine-grained Summarization Evaluation using LLMs](https://mlsanity.com/paper/2407.00908/) (2024)
- [Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators](https://mlsanity.com/paper/2504.15253/) (2025)
- [Self-Reflection in LLM Agents: Effects on Problem-Solving Performance](https://mlsanity.com/paper/2405.06682/) (2024)
- [No Free Lunch: Rethinking Internal Feedback for LLM Reasoning](https://mlsanity.com/paper/2506.17219/) (2025)
- [Towards Uncertainty-Aware Language Agent](https://mlsanity.com/paper/2401.14016/) (2024)
- [Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency](https://mlsanity.com/paper/2502.04964/) (2025)
- [TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning](https://mlsanity.com/paper/2506.10380/) (2025)
- [Uncertainty-Aware Step-wise Verification with Generative Reward Models](https://mlsanity.com/paper/2502.11250/) (2025)
- [An Empirical Analysis of Uncertainty in Large Language Model Evaluations](https://mlsanity.com/paper/2502.10709/) (2025)
- [Exploring LLM Agents for Cleaning Tabular Machine Learning Datasets](https://mlsanity.com/paper/2503.06664/) (2025)
- [Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Heads](https://mlsanity.com/paper/2505.20045/) (2025)
- [Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement](https://mlsanity.com/paper/2508.04025/) (2025)
- [SAUP: Situation Awareness Uncertainty Propagation on LLM Agent](https://mlsanity.com/paper/2412.01033/) (2024)
- [Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning](https://mlsanity.com/paper/2502.19652/) (2025)
- [MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification](https://mlsanity.com/paper/2412.04494/) (2024)
- [Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing](https://mlsanity.com/paper/2502.20592/) (2025)
- [The Need for Guardrails with Large Language Models in Medical Safety-Critical Settings: An Artificial Intelligence Application in the Pharmacovigilance Ecosystem](https://mlsanity.com/paper/2407.18322/) (2024)
- [Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges](https://mlsanity.com/paper/2508.00217/) (2025)

## Built on by (most cited first, 10 of 10)

- [Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents](https://mlsanity.com/paper/2602.02050/) (2026)
- [Calibration-Aware Policy Optimization for Reasoning LLMs](https://mlsanity.com/paper/2604.12632/) (2026)
- [Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents](https://mlsanity.com/paper/2605.09698/) (2026)
- [Towards Label-Free Biological Reasoning Synthetic Dataset Creation via Uncertainty Filtering](https://mlsanity.com/paper/2510.05871/) (2025)
- [From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models](https://mlsanity.com/paper/2601.15690/) (2026)
- [MechPert: Mechanistic Consensus as an Inductive Bias for Unseen Perturbation Prediction](https://mlsanity.com/paper/2602.13791/) (2026)
- [Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carrier](https://mlsanity.com/paper/2606.20662/) (2026)
- [Measuring Black-Box Confidence via Reasoning Trajectories: Geometry, Coverage, and Verbalization](https://mlsanity.com/paper/2605.06308/) (2026)
- [Semantic Uncertainty-Guided Orchestration in Hierarchical Multi-Agent Systems](https://mlsanity.com/paper/2608.14707/) (2026)
- [Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study](https://mlsanity.com/paper/2609.07395/) (2026)

## Abstract (arXiv)

Large language model (LLM) agents are increasingly deployed in structured biomedical data environments, yet they often produce fluent but overconfident outputs when reasoning over complex multi-table data. We introduce an uncertainty-aware agent for query-conditioned multi-table summarization that leverages two complementary signals: (i) retrieval uncertainty--entropy over multiple table-selection rollouts--and (ii) summary uncertainty--combining self-consistency and perplexity. Summary uncertainty is incorporated into reinforcement learning (RL) with Group Relative Policy Optimization (GRPO), while both retrieval and summary uncertainty guide inference-time filtering and support the construction of higher-quality synthetic datasets. On multi-omics benchmarks, our approach improves factuality and calibration, nearly tripling correct and useful claims per summary (3.0(rightarrow)8.4 internal; 3.6(rightarrow)9.9 cancer multi-omics) and substantially improving downstream survival prediction (C-index 0.32(rightarrow)0.63). These results demonstrate that uncertainty can serve as a control signal--enabling agents to abstain, communicate confidence, and become more reliable tools for complex structured-data environments.