# It Takes Two to Tango: Towards Theory of AI's Mind

arXiv: https://arxiv.org/abs/1704.00717 · HTML: https://mlsanity.com/paper/1704.00717/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2017.

## Built on by (most cited first, 11 of 11)

- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- [MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks](https://mlsanity.com/paper/2109.06275/) (2021)
- [Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models](https://mlsanity.com/paper/2310.19619/) (2023)
- [The Impact of Imperfect XAI on Human-AI Decision-Making](https://mlsanity.com/paper/2307.13566/) (2023)
- [Unpacking Large Language Models with Conceptual Consistency](https://mlsanity.com/paper/2209.15093/) (2022)
- [Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools](https://mlsanity.com/paper/2303.00192/) (2023)
- [Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflection](https://mlsanity.com/paper/2501.15355/) (2025)
- [Improving Human Sequential Decision-Making with Reinforcement Learning](https://mlsanity.com/paper/2108.08454/) (2021)
- [Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes](https://mlsanity.com/paper/2204.07693/) (2022)
- [Agents meet OKR: An Object and Key Results Driven Agent System with Hierarchical Self-Collaboration and Self-Evaluation](https://mlsanity.com/paper/2311.16542/) (2023)
- [Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models](https://mlsanity.com/paper/2103.14712/) (2021)

## Abstract (arXiv)

Theory of Mind is the ability to attribute mental states (beliefs, intents, knowledge, perspectives, etc.) to others and recognize that these mental states may differ from one's own. Theory of Mind is critical to effective communication and to teams demonstrating higher collective performance. To effectively leverage the progress in Artificial Intelligence (AI) to make our lives more productive, it is important for humans and AI to work well together in a team. Traditionally, there has been much emphasis on research to make AI more accurate, and (to a lesser extent) on having it better understand human intentions, tendencies, beliefs, and contexts. The latter involves making AI more human-like and having it develop a theory of our minds. In this work, we argue that for human-AI teams to be effective, humans must also develop a theory of AI's mind (ToAIM) - get to know its strengths, weaknesses, beliefs, and quirks. We instantiate these ideas within the domain of Visual Question Answering (VQA). We find that using just a few examples (50), lay people can be trained to better predict responses and oncoming failures of a complex VQA model. We further evaluate the role existing explanation (or interpretability) modalities play in helping humans build ToAIM. Explainable AI has received considerable scientific and popular attention in recent times. Surprisingly, we find that having access to the model's internal states - its confidence in its top-k predictions, explicit or implicit attention maps which highlight regions in the image (and words in the question) the model is looking at (and listening to) while answering a question about an image - do not help people better predict its behavior.