# Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?

arXiv: https://arxiv.org/abs/2401.05302 · HTML: https://mlsanity.com/paper/2401.05302/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 68 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Builds on (most cited first, 29 of 29)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [Deception Abilities Emerged in Large Language Models](https://mlsanity.com/paper/2307.16513/) (2023)
- [Event2Mind: Commonsense Inference on Events, Intents, and Reactions](https://mlsanity.com/paper/1805.06939/) (2018)
- [Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation](https://mlsanity.com/paper/2006.14804/) (2020)
- [Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations](https://mlsanity.com/paper/2002.01080/) (2020)
- [Explicability? Legibility? Predictability? Transparency? Privacy? Security? The Emerging Landscape of Interpretable Agent Behavior](https://mlsanity.com/paper/1811.09722/) (2018)
- [Plan Explicability and Predictability for Robot Task Planning](https://mlsanity.com/paper/1511.08158/) (2015)
- [Machine Common Sense Concept Paper](https://mlsanity.com/paper/1810.07528/) (2018)
- [Using Knowledge-Embedded Attention to Augment Pre-trained Language Models for Fine-Grained Emotion Recognition](https://mlsanity.com/paper/2108.00194/) (2021)
- [Signaling Friends and Head-Faking Enemies Simultaneously: Balancing Goal Obfuscation and Goal Legibility](https://mlsanity.com/paper/1905.10672/) (2019)
- [Symbol Guided Hindsight Priors for Reward Learning from Human Preferences](https://mlsanity.com/paper/2210.09151/) (2022)
- [Balancing Explicability and Explanation in Human-Aware Planning](https://mlsanity.com/paper/1708.00543/) (2017)
- [EmotionX-KU: BERT-Max based Contextual Emotion Classifier](https://mlsanity.com/paper/1906.11565/) (2019)
- [A Mental-Model Centric Landscape of Human-AI Symbiosis](https://mlsanity.com/paper/2202.09447/) (2022)
- [A Mental Model Based Framework of Trust](https://mlsanity.com/paper/2301.12569/) (2023)
- [Planning for Proactive Assistance in Environments with Partial Observability](https://mlsanity.com/paper/2105.00525/) (2021)

## Built on by (most cited first, 22 of 22)

- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks](https://mlsanity.com/paper/2402.01817/) (2024)
- [MuMA-ToM: Multi-modal Multi-Agent Theory of Mind](https://mlsanity.com/paper/2408.12574/) (2024)
- [Why human-AI relationships need socioaffective alignment](https://mlsanity.com/paper/2502.02528/) (2025)
- [Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning](https://mlsanity.com/paper/2405.20625/) (2024)
- [Language Models Represent Beliefs of Self and Others](https://mlsanity.com/paper/2402.18496/) (2024)
- [Theory of Mind in Large Language Models: Assessment and Enhancement](https://mlsanity.com/paper/2505.00026/) (2025)
- [Evaluating Large Language Models with Psychometrics](https://mlsanity.com/paper/2406.17675/) (2024)
- [On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models](https://mlsanity.com/paper/2405.13966/) (2024)
- [Perceptions of Linguistic Uncertainty by Language Models and Humans](https://mlsanity.com/paper/2407.15814/) (2024)
- [Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2405.15194/) (2024)
- [Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models](https://mlsanity.com/paper/2503.15491/) (2025)
- [Logic Contrastive Reasoning with Lightweight Large Language Model for Math Word Problems](https://mlsanity.com/paper/2409.00131/) (2024)
- [Plan-and-Act using Large Language Models for Interactive Agreement](https://mlsanity.com/paper/2504.01252/) (2025)
- [Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospects](https://mlsanity.com/paper/2505.05318/) (2025)
- [Are Large Language Models Aligned with People's Social Intuitions for Human-Robot Interactions?](https://mlsanity.com/paper/2403.05701/) (2024)
- [SocialNLI: A Dialogue-Centric Social Inference Dataset](https://mlsanity.com/paper/2510.05458/) (2025)
- [Trust in Vision-Language Models: Insights from a Participatory User Workshop](https://mlsanity.com/paper/2511.13458/) (2025)
- [Few-Shot Inference of Human Perceptions of Robot Performance in Social Navigation Scenarios](https://mlsanity.com/paper/2512.16019/) (2025)
- [Theory of Mind for Explainable Human-Robot Interaction](https://mlsanity.com/paper/2512.23482/) (2025)
- [CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs](https://mlsanity.com/paper/2603.11915/) (2026)
- [Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models](https://mlsanity.com/paper/2603.24484/) (2026)

## Abstract (arXiv)

Large Language Models have shown exceptional generative abilities in various natural language and generation tasks. However, possible anthropomorphization and leniency towards failure cases have propelled discussions on emergent abilities of Large Language Models especially on Theory of Mind (ToM) abilities in Large Language Models. While several false-belief tests exists to verify the ability to infer and maintain mental models of another entity, we study a special application of ToM abilities that has higher stakes and possibly irreversible consequences : Human Robot Interaction. In this work, we explore the task of Perceived Behavior Recognition, where a robot employs a Large Language Model (LLM) to assess the robot's generated behavior in a manner similar to human observer. We focus on four behavior types, namely - explicable, legible, predictable, and obfuscatory behavior which have been extensively used to synthesize interpretable robot behaviors. The LLMs goal is, therefore to be a human proxy to the agent, and to answer how a certain agent behavior would be perceived by the human in the loop, for example "Given a robot's behavior X, would the human observer find it explicable?". We conduct a human subject study to verify that the users are able to correctly answer such a question in the curated situations (robot setting and plan) across five domains. A first analysis of the belief test yields extremely positive results inflating ones expectations of LLMs possessing ToM abilities. We then propose and perform a suite of perturbation tests which breaks this illusion, i.e. Inconsistent Belief, Uninformative Context and Conviction Test. We conclude that, the high score of LLMs on vanilla prompts showcases its potential use in HRI settings, however to possess ToM demands invariance to trivial or irrelevant perturbations in the context which LLMs lack.