# When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs

arXiv: https://arxiv.org/abs/2602.06286 · HTML: https://mlsanity.com/paper/2602.06286/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Calibrating Large Language Models Using Their Generations Only](https://mlsanity.com/paper/2403.05973/) (2024)
- [Large Language Models Assume People are More Rational than We Really are](https://mlsanity.com/paper/2406.17055/) (2024)
- [Incoherent Probability Judgments in Large Language Models](https://mlsanity.com/paper/2401.16646/) (2024)
- [Standards for Belief Representations in LLMs](https://mlsanity.com/paper/2405.21030/) (2024)
- [Evaluating language models as risk scores](https://mlsanity.com/paper/2407.14614/) (2024)
- [pgmpy: A Python Toolkit for Bayesian Networks](https://mlsanity.com/paper/2304.08639/) (2023)
- [Eliciting the Priors of Large Language Models using Iterated In-Context Learning](https://mlsanity.com/paper/2406.01860/) (2024)
- [Calibrating Verbalized Probabilities for Large Language Models](https://mlsanity.com/paper/2410.06707/) (2024)
- [Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefs](https://mlsanity.com/paper/2504.13644/) (2025)

## Built on by (most cited first, 10 of 10)

- [Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theorems](https://mlsanity.com/paper/2608.05015/) (2026)
- [Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces](https://mlsanity.com/paper/2609.37470/) (2026)
- [Reasonably reasoning AI agents can avoid game-theoretic failures in zero-shot, provably](https://mlsanity.com/paper/2603.18563/) (2026)
- [High-Stakes Decisions with Language Models: Insights from Emergency Triage](https://mlsanity.com/paper/2608.01361/) (2026)
- [When Policies Change Probabilities: Modular Decision-Making for LLM Code Review](https://mlsanity.com/paper/2608.02677/) (2026)
- [Beliefs and Behavior in Language Models](https://mlsanity.com/paper/2609.07943/) (2026)
- [Preference Reasoning under Indeterminacy in Large Language Models](https://mlsanity.com/paper/2608.18631/) (2026)
- [LLM sequential decision making under uncertainty in biochemical domains](https://mlsanity.com/paper/2609.33061/) (2026)
- [Diagnosing and Improving Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2609.38005/) (2026)
- [Training Language Models To Be Coherent Decision-Makers](https://mlsanity.com/paper/2610.09164/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in high-stakes settings where good decisions require forming beliefs over the probability of unknown outcomes. However, it is unclear whether LLMs act as if they hold coherent beliefs when making decisions, or if so, how we could validate models' reports of such beliefs. We propose a decision-theoretic framework that elicits both probability judgments and decisions from an agent and tests their mutual consistency. Formally, our methods characterize whether it is possible for the actions to be produced by a ``near-rational" decision maker who holds the elicited probability as their true belief. We show that, perhaps surprisingly, this formalization implies empirically testable conditions even without any assumption about the agent's utility function. Applying our framework to stylized clinical diagnosis tasks, we find that models' reported beliefs are demonstrably imperfect summaries of the information revealed in their decisions, but that the discrepancies are small for the strongest models.