mlsanityData to 14 Oct 2026

MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks

arXiv 2109.06275 ↗First posted 13 Sep 2021cs.AI cs.CL cs.CVcited by 62

Not on the map. Why?

Abstract (arXiv)

An ideal integration of autonomous agents in a human world implies that they are able to collaborate on human terms. In particular, theory of mind plays an important role in maintaining common ground during human collaboration and communication. To enable theory of mind modeling in situated interactions, we introduce a fine-grained dataset of collaborative tasks performed by pairs of human subjects in the 3D virtual blocks world of Minecraft. It provides information that captures partners' beliefs of the world and of each other as an interaction unfolds, bringing abundant opportunities to study human collaborative behaviors in situated language communication. As a first step towards our goal of developing embodied AI agents able to infer belief states of collaborative partners in situ, we build and present results on computational models for several theory of mind tasks.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 11 of its 33 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

11 of 33 references.

References · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  4. 2018Machine Theory of Mindcited by 226
  5. 2019Executing Instructions in Situated Collaborative Interactionscited by 69
  6. 2019A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Contextcited by 35
  7. 2020RMM: A Recursive Mental Model for Dialog Navigationcited by 32
  8. 2021CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Modelscited by 20
  9. 2017Learning Interpretable Spatial Operations in a Rich 3D Blocks Worldcited by 15
  10. 2017It Takes Two to Tango: Towards Theory of AI's Mindcited by 11
  11. 2019An Annotated Corpus of Reference Resolution for Interpreting Common Groundingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 15; 2024: 14; 2025: 15; 2026: 9.

62 papers.

Sort:

By area:

  1. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  2. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  3. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
  4. 2023How FaR Are Large Language Models From Agents with Theory-of-Mind?cited by 76
  5. 2023Learning to Model the World with Languagecited by 68
  6. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 60
  7. 2023Grounding Gaps in Language Model Generationscited by 47
  8. 2023Towards A Holistic Landscape of Situated Theory of Mind in Large Language Modelscited by 32
  9. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 29
  10. 2025Theory of Mind in Large Language Models: Assessment and Enhancementcited by 25
  11. 2022I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragonscited by 22
  12. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 19
38 more
  1. 2023MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logiccited by 18
  2. 2023Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?cited by 18
  3. 2024Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Taskcited by 17
  4. 2025SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactionscited by 16
  5. 2022Learning to Mediate Disparities Towards Pragmatic Communicationcited by 14
  6. 2022Dialog Acts for Task-Driven Embodied Agentscited by 13
  7. 2023Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructionscited by 13
  8. 2025Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflectioncited by 13
  9. 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
  10. 2023Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategycited by 12
  11. 2022Werewolf Among Us: A Multimodal Dataset for Modeling Persuasion Behaviors in Social Deduction Gamescited by 11
  12. 2024CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Modelscited by 11
  13. 2024Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Futurecited by 11
  14. 2024Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Groundcited by 10
  15. 2024Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisitioncited by 10
  16. 2025AssistanceZero: Scalably Solving Assistance Gamescited by 10
  17. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
  18. 2022Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructionscited by 9
  19. 2023Towards Collaborative Plan Acquisition through Theory of Mind Modeling in Situated Dialoguecited by 9
  20. 2024Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Unitscited by 9
  21. 2023What A Situated Language-Using Agent Must be Able to Do: A Top-Down Analysiscited by 8
  22. 2024Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Modelscited by 8
  23. 2024Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactionscited by 6
  24. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 6
  25. 2022Evaluating Multimodal Interactive Agentscited by 5
  26. 2025Referential ambiguity and clarification requests: comparing human and LLM behaviourcited by 5
  27. 2025ToM-SSI: Evaluating Theory of Mind in Situated Social Interactionscited by 5
  28. 2025TactfulToM: Do LLMs Have the Theory of Mind Ability to Understand White Lies?cited by 5
  29. 2023Defining a New NLP Playgroundcited by 4
  30. 2025Understanding Common Ground Misalignment in Goal-Oriented Dialog: A Case-Study with Ubuntu Chat Logscited by 4
  31. 2023Towards a Neural Era in Dialogue Management for Collaboration: A Literature Surveycited by 3
  32. 2025The Yokai Learning Environment: Tracking Beliefs Over Space and Timecited by 3
  33. 2025Communication and Verification in LLM Agents towards Collaboration under Information Asymmetrycited by 3
  34. 2025Characterizing Language Use in a Collaborative Situated Gamecited by 3
  35. 2024Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programmingcited by 2
  36. 2025MDC-R: The Minecraft Dialogue Corpus with Referencecited by 2
  37. 2026DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectoriescited by 2
  38. 2026Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Taskscited by 2

50 most cited of 62 All 62 (JSON)