# Explore and Explain: Self-supervised Navigation and Recounting

arXiv: https://arxiv.org/abs/2007.07268 · HTML: https://mlsanity.com/paper/2007.07268/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 14 of 14)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Retrieval-Augmented Transformer for Image Captioning](https://mlsanity.com/paper/2207.13162/) (2022)
- [Distilling Internet-Scale Vision-Language Models into Embodied Agents](https://mlsanity.com/paper/2301.12507/) (2023)
- [Focus on Impact: Indoor Exploration with Intrinsic Motivation](https://mlsanity.com/paper/2109.08521/) (2021)
- [Out of the Box: Embodied Navigation in the Real World](https://mlsanity.com/paper/2105.05873/) (2021)
- [Embodied Agents for Efficient Exploration and Smart Scene Description](https://mlsanity.com/paper/2301.07150/) (2023)
- [Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents](https://mlsanity.com/paper/2412.11484/) (2024)
- [Spot the Difference: A Novel Task for Embodied Agents in Changing Environments](https://mlsanity.com/paper/2204.08502/) (2022)
- [Learning to Select: A Fully Attentive Approach for Novel Object Captioning](https://mlsanity.com/paper/2106.01424/) (2021)
- [Explore and Tell: Embodied Visual Captioning in 3D Environments](https://mlsanity.com/paper/2308.10447/) (2023)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [Embodied Navigation at the Art Gallery](https://mlsanity.com/paper/2204.09069/) (2022)
- [Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptions](https://mlsanity.com/paper/2504.08531/) (2025)
- [Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning](https://mlsanity.com/paper/2603.24257/) (2026)

## Abstract (arXiv)

Embodied AI has been recently gaining attention as it aims to foster the development of autonomous and intelligent agents. In this paper, we devise a novel embodied setting in which an agent needs to explore a previously unknown environment while recounting what it sees during the path. In this context, the agent needs to navigate the environment driven by an exploration goal, select proper moments for description, and output natural language descriptions of relevant objects and scenes. Our model integrates a novel self-supervised exploration module with penalty, and a fully-attentive captioning model for explanation. Also, we investigate different policies for selecting proper moments for explanation, driven by information coming from both the environment and the navigation. Experiments are conducted on photorealistic environments from the Matterport3D dataset and investigate the navigation and explanation capabilities of the agent as well as the role of their interactions.