# Violation of Expectation via Metacognitive Prompting Reduces Theory of Mind Prediction Error in Large Language Models

arXiv: https://arxiv.org/abs/2310.06983 · HTML: https://mlsanity.com/paper/2310.06983/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Are Emergent Abilities in Large Language Models just In-Context Learning?](https://mlsanity.com/paper/2309.01809/) (2023)
- [Could a Large Language Model be Conscious?](https://mlsanity.com/paper/2303.07103/) (2023)
- [Boosting Theory-of-Mind Performance in Large Language Models via Prompting](https://mlsanity.com/paper/2304.11490/) (2023)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)

## Built on by (most cited first, 4 of 4)

- [MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems](https://mlsanity.com/paper/2505.18943/) (2025)
- [Can Generative Agents Predict Emotion?](https://mlsanity.com/paper/2402.04232/) (2024)
- [Amico: An Event-Driven Modular Framework for Persistent and Embedded Autonomy](https://mlsanity.com/paper/2507.14513/) (2025)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)

## Abstract (arXiv)

Recent research shows that Large Language Models (LLMs) exhibit a compelling level of proficiency in Theory of Mind (ToM) tasks. This ability to impute unobservable mental states to others is vital to human social cognition and may prove equally important in principal-agent relations between individual humans and Artificial Intelligences (AIs). In this paper, we explore how a mechanism studied in developmental psychology known as Violation of Expectation (VoE) can be implemented to reduce errors in LLM prediction about users by leveraging emergent ToM affordances. And we introduce a textitmetacognitive prompting framework to apply VoE in the context of an AI tutor. By storing and retrieving facts derived in cases where LLM expectation about the user was violated, we find that LLMs are able to learn about users in ways that echo theories of human learning. Finally, we discuss latent hazards and augmentative opportunities associated with modeling user psychology and propose ways to mitigate risk along with possible directions for future inquiry.