# Towards Embodied Scene Description

arXiv: https://arxiv.org/abs/2004.14638 · HTML: https://mlsanity.com/paper/2004.14638/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 6 of 6)

- [Survey on reinforcement learning for language processing](https://mlsanity.com/paper/2104.05565/) (2021)
- [Internally Rewarded Reinforcement Learning](https://mlsanity.com/paper/2302.00270/) (2023)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [Self-Explainable Affordance Learning with Embodied Caption](https://mlsanity.com/paper/2404.05603/) (2024)
- [Explore and Tell: Embodied Visual Captioning in 3D Environments](https://mlsanity.com/paper/2308.10447/) (2023)
- [Apple: Toward General Active Perception via Reinforcement Learning](https://mlsanity.com/paper/2505.06182/) (2025)

## Abstract (arXiv)

Embodiment is an important characteristic for all intelligent agents (creatures and robots), while existing scene description tasks mainly focus on analyzing images passively and the semantic understanding of the scenario is separated from the interaction between the agent and the environment. In this work, we propose the Embodied Scene Description, which exploits the embodiment ability of the agent to find an optimal viewpoint in its environment for scene description tasks. A learning framework with the paradigms of imitation learning and reinforcement learning is established to teach the intelligent agent to generate corresponding sensorimotor activities. The proposed framework is tested on both the AI2Thor dataset and a real world robotic platform demonstrating the effectiveness and extendability of the developed method.