mlsanityData to 7 Oct 2026

When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs

arXiv 2602.06286 ↗First posted 6 Feb 2026cs.AIcited by 8

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in high-stakes settings where good decisions require forming beliefs over the probability of unknown outcomes. However, it is unclear whether LLMs act as if they hold coherent beliefs when making decisions, or if so, how we could validate models' reports of such beliefs. We propose a decision-theoretic framework that elicits both probability judgments and decisions from an agent and tests their mutual consistency. Formally, our methods characterize whether it is possible for the actions to be produced by a ``near-rational" decision maker who holds the elicited probability as their true belief. We show that, perhaps surprisingly, this formalization implies empirically testable conditions even without any assumption about the agent's utility function. Applying our framework to stylized clinical diagnosis tasks, we find that models' reported beliefs are demonstrably imperfect summaries of the information revealed in their decisions, but that the discrepancies are small for the strongest models.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 17 of its 47 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

17 of 47 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  3. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  5. 2022Using cognitive psychology to understand GPT-3cited by 389
  6. 2025Open Problems in Mechanistic Interpretabilitycited by 173
  7. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 145
  8. 2024Cognitive Bias in Decision-Making with LLMscited by 104
  9. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  10. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  11. 2024Standards for Belief Representations in LLMscited by 30
  12. 2024Incoherent Probability Judgments in Large Language Modelscited by 29
5 more
  1. 2024Evaluating language models as risk scorescited by 24
  2. 2023pgmpy: A Python Toolkit for Bayesian Networkscited by 21
  3. 2024Eliciting the Priors of Large Language Models using Iterated In-Context Learningcited by 19
  4. 2024Calibrating Verbalized Probabilities for Large Language Modelscited by 9
  5. 2025Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theoremscited by 3
  2. 2026Reasonably reasoning AI agents can avoid game-theoretic failures in zero-shot, provablycited by 1
  3. 2026When Policies Change Probabilities: Modular Decision-Making for LLM Code Reviewcited by 1
  4. 2026Beliefs and Behavior in Language Modelscited by 1
  5. 2026High-Stakes Decisions with Language Models: Insights from Emergency Triagecited by 0
  6. 2026Preference Reasoning under Indeterminacy in Large Language Modelscited by 0
  7. 2026Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfacescited by 0
  8. 2026Diagnosing and Improving Probabilistic Reasoning in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs
  2. 2026LLMs Should Not Yet Be Credited with Decision Explanation
  3. 2025Do Large Language Models Know What They Are Capable Of?
  4. 2026Before Agents Decide: Epistemic Action in LLM-Based Systems
  5. 2026Reasoning aligns language models to human cognition
  6. 2026From Plausible to Actionable: A Position on LLM Self-Explanations
  7. 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions
  8. 2025The simulation of judgment in LLMs