# SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency

arXiv: https://arxiv.org/abs/2112.01001 · HTML: https://mlsanity.com/paper/2112.01001/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 69 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Billion-scale semi-supervised learning for image classification](https://mlsanity.com/paper/1905.00546/) (2019)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [End-to-end Learning of Action Detection from Frame Glimpses in Videos](https://mlsanity.com/paper/1511.06984/) (2015)
- [Learning Active Learning from Data](https://mlsanity.com/paper/1703.03365/) (2017)
- [A Dataset for Developing and Benchmarking Active Vision](https://mlsanity.com/paper/1702.08272/) (2017)
- [Semantic Curiosity for Active Visual Learning](https://mlsanity.com/paper/2006.09367/) (2020)
- [Learning image representations tied to ego-motion](https://mlsanity.com/paper/1505.02206/) (2015)

## Built on by (most cited first, 30 of 48)

- [ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2304.03047/) (2023)
- [CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory](https://mlsanity.com/paper/2210.05663/) (2022)
- [Habitat-Matterport 3D Semantics Dataset](https://mlsanity.com/paper/2210.05633/) (2022)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation](https://mlsanity.com/paper/2404.06609/) (2024)
- [Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2210.07506/) (2022)
- [Explore until Confident: Efficient Exploration for Embodied Question Answering](https://mlsanity.com/paper/2403.15941/) (2024)
- [Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes](https://mlsanity.com/paper/2509.06266/) (2025)
- [GaussNav: Gaussian Splatting for Visual Navigation](https://mlsanity.com/paper/2403.11625/) (2024)
- [3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification](https://mlsanity.com/paper/2212.00338/) (2022)
- [Embodied AI: From LLMs to World Models](https://mlsanity.com/paper/2509.20021/) (2025)
- [Interactron: Embodied Adaptive Object Detection](https://mlsanity.com/paper/2202.00660/) (2022)
- [Ask4Help: Learning to Leverage an Expert for Embodied Tasks](https://mlsanity.com/paper/2211.09960/) (2022)
- [Object Goal Navigation with Recursive Implicit Maps](https://mlsanity.com/paper/2308.05602/) (2023)
- [AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splatting](https://mlsanity.com/paper/2502.04981/) (2025)
- [FrontierNet: Learning Visual Cues to Explore](https://mlsanity.com/paper/2501.04597/) (2025)
- [Annotator: A Generic Active Learning Baseline for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2310.20293/) (2023)
- [AutoNeRF: Training Implicit Scene Representations with Autonomous Agents](https://mlsanity.com/paper/2304.11241/) (2023)
- [An Informative Path Planning Framework for Active Learning in UAV-based Semantic Mapping](https://mlsanity.com/paper/2302.03347/) (2023)
- [Palm up: Playing in the Latent Manifold for Unsupervised Pretraining](https://mlsanity.com/paper/2210.10913/) (2022)
- [Self-supervised 3D Semantic Representation Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2201.10788/) (2022)
- [A Simple Approach for Visual Rearrangement: 3D Mapping and Semantic Search](https://mlsanity.com/paper/2206.13396/) (2022)
- [Semi-Supervised Active Learning for Semantic Segmentation in Unknown Environments Using Informative Path Planning](https://mlsanity.com/paper/2312.04402/) (2023)
- [Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation](https://mlsanity.com/paper/2409.14891/) (2024)
- [SCIM: Simultaneous Clustering, Inference, and Mapping for Open-World Semantic Scene Understanding](https://mlsanity.com/paper/2206.10670/) (2022)
- [Self-Supervised Object Goal Navigation with In-Situ Finetuning](https://mlsanity.com/paper/2212.05923/) (2022)
- [Masked Path Modeling for Vision-and-Language Navigation](https://mlsanity.com/paper/2305.14268/) (2023)
- [ALP: Action-Aware Embodied Learning for Perception](https://mlsanity.com/paper/2306.10190/) (2023)
- [Embodied Active Defense: Leveraging Recurrent Feedback to Counter Adversarial Patches](https://mlsanity.com/paper/2404.00540/) (2024)
- [MoDA: Map style transfer for self-supervised Domain Adaptation of embodied agents](https://mlsanity.com/paper/2211.15992/) (2022)

## Abstract (arXiv)

In this paper, we explore how we can build upon the data and models of Internet images and use them to adapt to robot vision without requiring any extra labels. We present a framework called Self-supervised Embodied Active Learning (SEAL). It utilizes perception models trained on internet images to learn an active exploration policy. The observations gathered by this exploration policy are labelled using 3D consistency and used to improve the perception model. We build and utilize 3D semantic maps to learn both action and perception in a completely self-supervised manner. The semantic map is used to compute an intrinsic motivation reward for training the exploration policy and for labelling the agent observations using spatio-temporal 3D consistency and label propagation. We demonstrate that the SEAL framework can be used to close the action-perception loop: it improves object detection and instance segmentation performance of a pretrained perception model by just moving around in training environments and the improved perception model can be used to improve Object Goal Navigation.