# Superficial Beliefs in LLM Decision-Making

arXiv: https://arxiv.org/abs/2606.11016 · HTML: https://mlsanity.com/paper/2606.11016/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 11 of 11)

- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Reasoning Models Don't Always Say What They Think](https://mlsanity.com/paper/2505.05410/) (2025)
- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life](https://mlsanity.com/paper/2410.02683/) (2024)
- [Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?](https://mlsanity.com/paper/2501.15463/) (2025)
- [Argumentative Large Language Models for Explainable and Contestable Claim Verification](https://mlsanity.com/paper/2405.02079/) (2024)
- [Standards for Belief Representations in LLMs](https://mlsanity.com/paper/2405.21030/) (2024)
- [Do LLMs have Consistent Values?](https://mlsanity.com/paper/2407.12878/) (2024)
- [Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?](https://mlsanity.com/paper/2506.00751/) (2025)
- [Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefs](https://mlsanity.com/paper/2504.13644/) (2025)
- [Argumentation for Explainable and Globally Contestable Decision Support with LLMs](https://mlsanity.com/paper/2603.14643/) (2026)

## Abstract (arXiv)

We ask whether large language models (LLMs) merely imitate rationales when choosing between two options, or whether their choices reflect a systematic underlying decision structure. Using synthetic binary decision settings in which models choose between profiles defined by graded attributes, we compare the attribute a model says mattered most with the attribute that best explains its choice under a behavioural model fit to prior decisions. The behavioural model predicts held-out choices well, showing that model behaviour is systematically related to the visible attributes rather than being random. However, direct self-reports and a separate score-based judge recover the behaviourally inferred driver only partially. The resulting picture is neither one of arbitrary behaviour nor one of fully articulated belief - outputs are structured enough to support prediction, but explicit reasons track the recovered driver only imperfectly. This qualitative pattern persists across prompt-order and sampling perturbations, alternative behavioural models, targeted occlusion analyses, and structurally varied decision settings. We interpret this as evidence for ``superficial belief'' in LLM decision-making: models behave as if guided by probabilistic local priorities over attributes, while having only limited verbal access to the attributes that drive their decisions.