# Are language models rational? The case of coherence norms and belief revision

arXiv: https://arxiv.org/abs/2406.03442 · HTML: https://mlsanity.com/paper/2406.03442/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 23 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 5 of 5)

- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty](https://mlsanity.com/paper/2401.06730/) (2024)
- [Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs](https://mlsanity.com/paper/2111.13654/) (2021)

## Built on by (most cited first, 10 of 10)

- [Belief Revision: The Adaptability of Large Language Models Reasoning](https://mlsanity.com/paper/2406.19764/) (2024)
- [Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?](https://mlsanity.com/paper/2406.19354/) (2024)
- [Teaching Models to Balance Resisting and Accepting Persuasion](https://mlsanity.com/paper/2410.14596/) (2024)
- [Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing](https://mlsanity.com/paper/2410.17194/) (2024)
- [Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance](https://mlsanity.com/paper/2407.07950/) (2024)
- [Calibrating Verbalized Confidence with Self-Generated Distractors](https://mlsanity.com/paper/2509.25532/) (2025)
- [RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models](https://mlsanity.com/paper/2504.11381/) (2025)
- [Language Models Identify Ambiguities and Exploit Loopholes](https://mlsanity.com/paper/2508.19546/) (2025)
- [Belief Graphs with Reasoning Zones: Structure, Dynamics, and Epistemic Activation](https://mlsanity.com/paper/2510.10042/) (2025)
- [Radical AI Interpretability](https://mlsanity.com/paper/2606.26523/) (2026)

## Abstract (arXiv)

Do norms of rationality apply to machine learning models, in particular language models? In this paper we investigate this question by focusing on a special subset of rational norms: coherence norms. We consider both logical coherence norms as well as coherence norms tied to the strength of belief. To make sense of the latter, we introduce the Minimal Assent Connection (MAC) and propose a new account of credence, which captures the strength of belief in language models. This proposal uniformly assigns strength of belief simply on the basis of model internal next token probabilities. We argue that rational norms tied to coherence do apply to some language models, but not to others. This issue is significant since rationality is closely tied to predicting and explaining behavior, and thus it is connected to considerations about AI safety and alignment, as well as understanding model behavior more generally.