mlsanityData to 14 Oct 2026

CaRT: Teaching LLM Agents to Know When They Know Enough

arXiv 2510.08517 ↗First posted 9 Oct 2025cs.AI cs.CL cs.LGcited by 7

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Many tasks require learned models to strategically gather relevant information over multiple rounds of interaction before actually acting on a task. Strategic information gathering requires models to know not only how to effectively acquire information, but also when to stop gathering information and make a decision, in order to avoid overthinking or getting derailed when acting. In this paper, we formalize this problem and introduce Counterfactuals and Reasoning for Termination (CaRT), an approach for teaching LLMs when to stop seeking information. To appropriately learn when to terminate, CaRT fine-tunes LLMs using counterfactual pairs of trajectories, one where termination is appropriate and a minimally modified version of the same trajectory where it is not. It trains the LLM to explain the rationale for the termination decision in either case via verbal reasoning, and imbues this capability into the base LLM via fine-tuning. We instantiate CaRT in two domains: interactive medical diagnosis and math problem solving. In both domains, we find that CaRT improves the efficiency of information gathering and task success rate compared to other fine-tuning methods.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 43 of its 64 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

43 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Adaptive Domain Modeling with Language Models: A Multi-Agent Approach to Task Planning24 Jun
  2. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planning18 Jun
  3. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents1 Nov

Other references · 40

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  5. 2025s1: Simple test-time scalingcited by 1,341
  6. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  7. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  8. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  10. 2025Training Language Models to Reason Efficientlycited by 247
  11. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  12. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
28 more
  1. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
  2. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 116
  3. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 98
  4. 2023Eliciting Human Preferences with Language Modelscited by 92
  5. 2024Can large language models explore in-context?cited by 76
  6. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 76
  7. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 69
  8. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  9. 2024Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Modelscited by 54
  10. 2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Modelscited by 54
  11. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  12. 2019Learning When-to-Treat Policiescited by 49
  13. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
  14. 2021Towards Robust Classification Model by Counterfactual and Invariant Data Generationcited by 30
  15. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  16. 2025Large Language Models are overconfident and amplify human biascited by 30
  17. 2023Primacy Effect of ChatGPTcited by 27
  18. 2025Reasoning as an Adaptive Defense for Safetycited by 23
  19. 2025Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoningcited by 19
  20. 2025Toward Efficient Exploration by Large Language Model Agentscited by 17
  21. 2025Language Agents for Hypothesis-driven Clinical Decision Making with Reinforcement Learningcited by 15
  22. 2023Data Augmentations for Improved (Large) Language Model Generalizationcited by 14
  23. 2024Empowering Language Models with Active Inquiry for Deeper Understandingcited by 11
  24. 2025Feedback-Aware Monte Carlo Tree Search for Efficient Information Seeking in Goal-Oriented Conversationscited by 6
  25. 2025Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Trainingcited by 5
  26. 2017Sample Efficient Policy Search for Optimal Stopping Domainscited by 4
  27. 2022Solving the optimal stopping problem with reinforcement learning: an application in financial option exercisecited by 3
  28. 2025SALAD: Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM-Driven Augmented Datacited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?cited by 4
  2. 2026EarlyEval: Cheaper Agent Evaluation via Early Outcome Predictioncited by 3
  3. 2026When Should Models Change Their Minds? Contextual Belief Management in Large Language Modelscited by 2
  4. 2026Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agentscited by 0
  5. 2026DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?cited by 0
  6. 2026CIPO: Counterfactual Imagination Policy Optimization for Adaptive Tool Granularity Selectioncited by 0
  7. 2026RIVO: Lightweight Adaptive Rerouting for Deep Research Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain
  2. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following
  3. 2025LLM Agents Beyond Utility: An Open-Ended Perspective
  4. 2026Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning
  5. 2025Thoughts without Thinking: Reconsidering the Explanatory Value of Chain-of-Thought Reasoning in LLMs through Agentic Pipelines
  6. 2026Statistical Early Stopping for Reasoning Models
  7. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMs
  8. 2025Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?