Playing games with knowledge: AI-Induced delusions need game theoretic interventions
Not on the map. Why?
Abstract (arXiv)
Conversational AI has a fundamental flaw as a knowledge interface: sycophantic chatbots induce epistemic entrenchment and delusional belief spirals even in rational agents. We propose the problem does not stem from the AI model, rooted instead in a systemic consequence of the paradigm shift from user-driven knowledge search to users and agents engaged in strategic, repeated-play communication. We formalize the problem as a Crawford-Sobel cheap talk game, where costless user signals induce a pooling equilibrium. Agents optimized for user satisfaction produce sycophantic strategies that provide identical reinforcement across user types with opposite epistemic incentives: exploratory ``Growth-seekers'' (θG) and confirmatory ``Validation-seekers'' (θV). Under repeated play, this identification failure creates a coordination trap -- analogous to a Prisoner's Dilemma -- where locally rational feedback loops drive users toward pathologically certain false beliefs. We propose an inference-time mechanism design intervention called an Epistemic Mediator that breaks this pooling equilibrium by introducing a costly signal (epistemic friction), forcing type revelation based on users' asymmetric cognitive costs for processing resistance. A key contribution is Belief Versioning, a git-inspired epistemic meta-memory system that stores healthy beliefs and rollbacks when validation-seeking resistance is detected. In simulation, this intervention achieves a separating equilibrium achieving a 48× differential in spiral rates while passing a learning preservation criterion), evidence that epistemic safety in AI is fundamentally a problem of strategic information environment design rather than simple model alignment.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 11 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
5 of 11 references.
References · 5
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2023Scalable AI Safety via Doubly-Efficient Debatecited by 57
- 2026Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesianscited by 21
- 2025BASIL: Bayesian Assessment of Sycophancy in LLMscited by 13
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Architecting Trust in Artificial Epistemic Agents
- 2026MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs
- 2026MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games
- 2026Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics
- 2026Deception and Communication in Autonomous Multi-Agent Systems: An Experimental Study with Among Us
- 2026Readable Minds: Emergent Theory-of-Mind-Like Behavior in LLM Poker Agents
- 2026Lying to Win: Assessing LLM Deception through Human-AI Games and Parallel-World Probing
- 2026QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents