# RMM: A Recursive Mental Model for Dialog Navigation

arXiv: https://arxiv.org/abs/2005.00728 · HTML: https://mlsanity.com/paper/2005.00728/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 30 of 32)

- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning](https://mlsanity.com/paper/2104.05832/) (2021)
- [DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following](https://mlsanity.com/paper/2202.13330/) (2022)
- [MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks](https://mlsanity.com/paper/2109.06275/) (2021)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [Alexa Arena: A User-Centric Interactive Platform for Embodied AI](https://mlsanity.com/paper/2303.01586/) (2023)
- [Adaptive Coordination in Social Embodied Rearrangement](https://mlsanity.com/paper/2306.00087/) (2023)
- [VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator](https://mlsanity.com/paper/2105.11589/) (2021)
- [A Framework for Learning to Request Rich and Contextually Useful Information from Humans](https://mlsanity.com/paper/2110.08258/) (2021)
- [Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue](https://mlsanity.com/paper/2210.04443/) (2022)
- [Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models](https://mlsanity.com/paper/2301.05149/) (2022)
- [NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation](https://mlsanity.com/paper/2510.16457/) (2025)
- [R2H: Building Multimodal Navigation Helpers that Respond to Help Requests](https://mlsanity.com/paper/2305.14260/) (2023)
- [The Future of Cognitive Strategy-enhanced Persuasive Dialogue Agents: New Perspectives and Trends](https://mlsanity.com/paper/2402.04631/) (2024)
- [ELBA: Learning by Asking for Embodied Visual Navigation and Task Completion](https://mlsanity.com/paper/2302.04865/) (2023)
- [Towards a Neural Era in Dialogue Management for Collaboration: A Literature Survey](https://mlsanity.com/paper/2307.09021/) (2023)
- [DialNav: Multi-turn Dialog Navigation with a Remote Guide](https://mlsanity.com/paper/2509.12894/) (2025)
- [Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigation](https://mlsanity.com/paper/2606.03175/) (2026)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [Extending Phrase Grounding with Pronouns in Visual Dialogues](https://mlsanity.com/paper/2210.12658/) (2022)
- [Partner Personas Generation for Diverse Dialogue Generation](https://mlsanity.com/paper/2111.13833/) (2021)
- [I2EDL: Interactive Instruction Error Detection and Localization](https://mlsanity.com/paper/2406.05080/) (2024)
- [Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling](https://mlsanity.com/paper/2306.12387/) (2023)

## Abstract (arXiv)

Language-guided robots must be able to both ask humans questions and understand answers. Much existing work focuses only on the latter. In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers. Inspired by theory of mind, we propose the Recursive Mental Model (RMM). The navigating agent models the guiding agent to simulate answers given candidate generated questions. The guiding agent in turn models the navigating agent to simulate navigation steps it would take to generate answers. We use the progress agents make towards the goal as a reinforcement learning reward signal to directly inform not only navigation actions, but also both question and answer generation. We demonstrate that RMM enables better generalization to novel environments. Interlocutor modelling may be a way forward for human-agent dialogue where robots need to both ask and answer questions.