# How FaR Are Large Language Models From Agents with Theory-of-Mind?

arXiv: https://arxiv.org/abs/2310.03051 · HTML: https://mlsanity.com/paper/2310.03051/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Language Models as Agent Models](https://mlsanity.com/paper/2212.01681/) (2022)
- [Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://mlsanity.com/paper/2306.00924/) (2023)
- [Evaluating Theory of Mind in Question Answering](https://mlsanity.com/paper/1808.09352/) (2018)
- [MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks](https://mlsanity.com/paper/2109.06275/) (2021)
- [Do Large Language Models know what humans know?](https://mlsanity.com/paper/2209.01515/) (2022)
- [I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons](https://mlsanity.com/paper/2212.10060/) (2022)

## Built on by (most cited first, 30 of 76)

- [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [Self-Discover: Large Language Models Self-Compose Reasoning Structures](https://mlsanity.com/paper/2402.03620/) (2024)
- [The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models](https://mlsanity.com/paper/2406.05761/) (2024)
- [MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming](https://mlsanity.com/paper/2310.06500/) (2023)
- [ToMBench: Benchmarking Theory of Mind in Large Language Models](https://mlsanity.com/paper/2402.15052/) (2024)
- [SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs](https://mlsanity.com/paper/2410.13648/) (2024)
- [OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2402.06044/) (2024)
- [Merlin:Empowering Multimodal LLMs with Foresight Minds](https://mlsanity.com/paper/2312.00589/) (2023)
- [CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update](https://mlsanity.com/paper/2312.10908/) (2023)
- [Language Models use Lookbacks to Track Beliefs](https://mlsanity.com/paper/2505.14685/) (2025)
- [Agents Thinking Fast and Slow: A Talker-Reasoner Architecture](https://mlsanity.com/paper/2410.08328/) (2024)
- [Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning](https://mlsanity.com/paper/2412.12175/) (2024)
- [Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models](https://mlsanity.com/paper/2310.19619/) (2023)
- [Language Models Represent Beliefs of Self and Others](https://mlsanity.com/paper/2402.18496/) (2024)
- [Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?](https://mlsanity.com/paper/2402.11597/) (2024)
- [Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Game](https://mlsanity.com/paper/2312.17515/) (2023)
- [InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context](https://mlsanity.com/paper/2406.12203/) (2024)
- [Theory of Mind in Large Language Models: Assessment and Enhancement](https://mlsanity.com/paper/2505.00026/) (2025)
- [Re-evaluating Theory of Mind evaluation in large language models](https://mlsanity.com/paper/2502.21098/) (2025)
- [Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models](https://mlsanity.com/paper/2407.06004/) (2024)
- [Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User Simulation](https://mlsanity.com/paper/2403.06769/) (2024)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [A Notion of Complexity for Theory of Mind via Discrete World Models](https://mlsanity.com/paper/2406.11911/) (2024)
- [Perceptions of Linguistic Uncertainty by Language Models and Humans](https://mlsanity.com/paper/2407.15814/) (2024)
- [Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models](https://mlsanity.com/paper/2412.17128/) (2024)
- [Evolving AI Collectives to Enhance Human Diversity and Enable Self-Regulation](https://mlsanity.com/paper/2402.12590/) (2024)
- [Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflection](https://mlsanity.com/paper/2501.15355/) (2025)
- [From Perceptions to Decisions: Wildfire Evacuation Decision Prediction with Behavioral Theory-informed LLMs](https://mlsanity.com/paper/2502.17701/) (2025)
- [Toward a Team of AI-made Scientists for Scientific Discovery from Gene Expression Data](https://mlsanity.com/paper/2402.12391/) (2024)

## Abstract (arXiv)

"Thinking is for Doing." Humans can infer other people's mental states from observations--an ability called Theory-of-Mind (ToM)--and subsequently act pragmatically on those inferences. Existing question answering benchmarks such as ToMi ask models questions to make inferences about beliefs of characters in a story, but do not test whether models can then use these inferences to guide their actions. We propose a new evaluation paradigm for large language models (LLMs): Thinking for Doing (T4D), which requires models to connect inferences about others' mental states to actions in social scenarios. Experiments on T4D demonstrate that LLMs such as GPT-4 and PaLM 2 seemingly excel at tracking characters' beliefs in stories, but they struggle to translate this capability into strategic action. Our analysis reveals the core challenge for LLMs lies in identifying the implicit inferences about mental states without being explicitly asked about as in ToMi, that lead to choosing the correct action in T4D. To bridge this gap, we introduce a zero-shot prompting framework, Foresee and Reflect (FaR), which provides a reasoning structure that encourages LLMs to anticipate future challenges and reason about potential actions. FaR boosts GPT-4's performance from 50% to 71% on T4D, outperforming other prompting methods such as Chain-of-Thought and Self-Ask. Moreover, FaR generalizes to diverse out-of-distribution story structures and scenarios that also require ToM inferences to choose an action, consistently outperforming other methods including few-shot in-context learning.