# Embodied Visual Active Learning for Semantic Segmentation

arXiv: https://arxiv.org/abs/2012.09503 · HTML: https://mlsanity.com/paper/2012.09503/

## Key facts

- Cited by 19 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 17 Dec 2020.

## Built on by (most cited first, 19 of 19)

- [Bird's-Eye-View Scene Graph for Vision-Language Navigation](https://mlsanity.com/paper/2308.04758/) (2023)
- [SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency](https://mlsanity.com/paper/2112.01001/) (2021)
- [Interactron: Embodied Adaptive Object Detection](https://mlsanity.com/paper/2202.00660/) (2022)
- [Learning Active Camera for Multi-Object Navigation](https://mlsanity.com/paper/2210.07505/) (2022)
- [Ask4Help: Learning to Leverage an Expert for Embodied Tasks](https://mlsanity.com/paper/2211.09960/) (2022)
- [Style Blind Domain Generalized Semantic Segmentation via Covariance Alignment and Semantic Consistence Contrastive Learning](https://mlsanity.com/paper/2403.06122/) (2024)
- [Evidential Active Recognition: Intelligent and Prudent Open-World Embodied Perception](https://mlsanity.com/paper/2311.13793/) (2023)
- [Unveiling and Mitigating Bias in Audio Visual Segmentation](https://mlsanity.com/paper/2407.16638/) (2024)
- [Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations](https://mlsanity.com/paper/2311.17938/) (2023)
- [Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation](https://mlsanity.com/paper/2410.01341/) (2024)
- [Exploring Probabilistic Modeling Beyond Domain Generalization for Semantic Segmentation](https://mlsanity.com/paper/2507.21367/) (2025)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [An Active and Contrastive Learning Framework for Fine-Grained Off-Road Semantic Segmentation](https://mlsanity.com/paper/2202.09002/) (2022)
- [Relative Difficulty Distillation for Semantic Segmentation](https://mlsanity.com/paper/2407.03719/) (2024)
- [SegmATRon: Embodied Adaptive Semantic Segmentation for Indoor Environment](https://mlsanity.com/paper/2310.12031/) (2023)
- [QueryAdapter: Rapid Adaptation of Vision-Language Models in Response to Natural Language Queries](https://mlsanity.com/paper/2502.18735/) (2025)
- [Embodied Learning for Lifelong Visual Perception](https://mlsanity.com/paper/2112.14084/) (2021)
- [Responsible Active Learning via Human-in-the-loop Peer Study](https://mlsanity.com/paper/2211.13587/) (2022)
- [Active Vision Reinforcement Learning under Limited Visual Observability](https://mlsanity.com/paper/2306.00975/) (2023)

## Abstract (arXiv)

We study the task of embodied visual active learning, where an agent is set to explore a 3d environment with the goal to acquire visual scene understanding by actively selecting views for which to request annotation. While accurate on some benchmarks, today's deep visual recognition pipelines tend to not generalize well in certain real-world scenarios, or for unusual viewpoints. Robotic perception, in turn, requires the capability to refine the recognition capabilities for the conditions where the mobile system operates, including cluttered indoor environments or poor illumination. This motivates the proposed task, where an agent is placed in a novel environment with the objective of improving its visual recognition capability. To study embodied visual active learning, we develop a battery of agents - both learnt and pre-specified - and with different levels of knowledge of the environment. The agents are equipped with a semantic segmentation network and seek to acquire informative views, move and explore in order to propagate annotations in the neighbourhood of those views, then refine the underlying segmentation network by online retraining. The trainable method uses deep reinforcement learning with a reward function that balances two competing objectives: task performance, represented as visual recognition accuracy, which requires exploring the environment, and the necessary amount of annotated data requested during active exploration. We extensively evaluate the proposed models using the photorealistic Matterport3D simulator and show that a fully learnt method outperforms comparable pre-specified counterparts, even when requesting fewer annotations.