Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents
On the map: LLM reasoning → Reasoning for agents, planning and code
also in 1 more
Abstract (arXiv)
Do LLM agents act on the reasoning they state? This question of process fidelity is central to LLM-based social simulation, yet hard to measure where no reference for correct behavior exists. We study it in a controlled setting: a Texas Poker simulator with a verifiable reference action for every decision by splitting the faithfulness gap into two steps: reasoning-to-conclusion (does the stated decision follow from the agent's own reasoning?) and conclusion-to-action (does the agent execute what it states?). The two steps behave very differently. Conclusion-to-action is reliable: inconsistency is 0.7% for Claude Haiku 4.5 and 1.4% for DeepSeek-Reasoner once the conclusion is read from an explicit tag, whereas free-text conclusion extraction reports 22-26%. Reasoning-to-conclusion is where fidelity frays, but not through a single dominant failure. In a step-level diagnostic the agent's errors split roughly evenly between bad inputs, borderline cases, and rule misapplication deriving a conclusion that contradicts the agent's own restated rule from inputs it estimated correctly. This composition is model-dependent: rule misapplication accounts for a third of Haiku's interpretable errors but only 8% of DeepSeek's. The one robust signal is directional: when an agent does misapply its own stated rule, it almost always (99.5% for Haiku) errs in the risk-averse direction. The override is partly hedging behavior, not a capability limit: instructing the agent to apply the rule mechanically halves the misapplication rate (13.9% to 6.8% of decisions) and raises adherence by eight points. Process-fidelity evaluation should therefore elicit machine-checkable conclusions and probe for directional biases rather than assume a single upstream failure mode, lest it conflate measurement noise with model behavior.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 8 of its 11 references that are in the map. First posted on arXiv 30 May 2026.
Builds on
8 of 11 references.
Closest ancestors · 2
Same direction, newest first
- 2026How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Use31 Jan
- 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents7 Jun
Other references · 6
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
- 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
- 2026M3-BENCH: Process-Aware Evaluation of LLM Agents' Social Behaviors in Mixed-Motive Gamescited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Project Ariadne: A Structural Causal Framework for Auditing Faithfulness in LLM Agents
- 2026Verify Before You Commit: Towards Faithful Reasoning in LLM Agents via Self-Auditing
- 2026Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes
- 2026Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation
- 2026The LLM Proposes, the Executive Disposes: A Self-Verifying Agent Instrument that Dissociates Commitment Drift from Binding Drift in Long-Horizon Agents
- 2026What People Almost Did: Evaluating LLM Social Simulations Beyond Behavioral Fit ↗
- 2025From the Fluency Fallacy to the Micro-to-Macro Validity Gap: Opportunities and Pitfalls of LLMs in Social Simulation
- 2026When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games