# Learning Semantic Traversability with Egocentric Video and Automated Annotation Strategy

arXiv: https://arxiv.org/abs/2406.02989 · HTML: https://mlsanity.com/paper/2406.02989/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [BADGR: An Autonomous Self-Supervised Learning-Based Navigation System](https://mlsanity.com/paper/2002.05700/) (2020)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion](https://mlsanity.com/paper/2308.12517/) (2023)
- [ViPlanner: Visual Semantic Imperative Learning for Local Navigation](https://mlsanity.com/paper/2310.00982/) (2023)
- [Fast Traversability Estimation for Wild Visual Navigation](https://mlsanity.com/paper/2305.08510/) (2023)
- [WayFAST: Navigation with Predictive Traversability in the Field](https://mlsanity.com/paper/2203.12071/) (2022)
- [V-STRONG: Visual Self-Supervised Traversability Learning for Off-road Navigation](https://mlsanity.com/paper/2312.16016/) (2023)
- [Find Your Own Way: Weakly-Supervised Segmentation of Path Proposals for Urban Autonomy](https://mlsanity.com/paper/1610.01238/) (2016)
- [Locomotion Policy Guided Traversability Learning using Volumetric Representations of Complex Environments](https://mlsanity.com/paper/2203.15854/) (2022)
- [Learning Off-Road Terrain Traversability with Self-Supervisions Only](https://mlsanity.com/paper/2305.18896/) (2023)
- [Self-Supervised Traversability Prediction by Learning to Reconstruct Safe Terrain](https://mlsanity.com/paper/2208.01329/) (2022)
- [Visual Representation Learning for Preference-Aware Path Planning](https://mlsanity.com/paper/2109.08968/) (2021)
- [STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience](https://mlsanity.com/paper/2309.15302/) (2023)
- [MEM: Multi-Modal Elevation Mapping for Robotics and Learning](https://mlsanity.com/paper/2309.16818/) (2023)
- [Learning Forward Dynamics Model and Informed Trajectory Sampler for Safe Quadruped Navigation](https://mlsanity.com/paper/2204.08647/) (2022)
- [Learning Semantics-Aware Locomotion Skills from Human Demonstration](https://mlsanity.com/paper/2206.13631/) (2022)
- [Learning Vehicle Dynamics from Cropped Image Patches for Robot Navigation in Unpaved Outdoor Terrains](https://mlsanity.com/paper/2309.02745/) (2023)

## Built on by (most cited first, 5 of 5)

- [Watch Your STEPP: Semantic Traversability Estimation using Pose Projected Features](https://mlsanity.com/paper/2501.17594/) (2025)
- [Scene-Agnostic Traversability Labeling and Estimation via a Multimodal Self-supervised Framework](https://mlsanity.com/paper/2508.18249/) (2025)
- [Learning Quadrupedal Locomotion for a Heavy Hydraulic Robot Using an Actuator Model](https://mlsanity.com/paper/2601.11143/) (2026)
- [SyNeT: Synthetic Negatives for Traversability Learning](https://mlsanity.com/paper/2602.00814/) (2026)
- [Long-Distance Real-World Navigation of the Legged-Wheeled Robot Go2-W Using Deep Reinforcement Learning](https://mlsanity.com/paper/2606.21387/) (2026)

## Abstract (arXiv)

For reliable autonomous robot navigation in urban settings, the robot must have the ability to identify semantically traversable terrains in the image based on the semantic understanding of the scene. This reasoning ability is based on semantic traversability, which is frequently achieved using semantic segmentation models fine-tuned on the testing domain. This fine-tuning process often involves manual data collection with the target robot and annotation by human labelers which is prohibitively expensive and unscalable. In this work, we present an effective methodology for training a semantic traversability estimator using egocentric videos and an automated annotation process. Egocentric videos are collected from a camera mounted on a pedestrian's chest. The dataset for training the semantic traversability estimator is then automatically generated by extracting semantically traversable regions in each video frame using a recent foundation model in image segmentation and its prompting technique. Extensive experiments with videos taken across several countries and cities, covering diverse urban scenarios, demonstrate the high scalability and generalizability of the proposed annotation method. Furthermore, performance analysis and real-world deployment for autonomous robot navigation showcase that the trained semantic traversability estimator is highly accurate, able to handle diverse camera viewpoints, computationally light, and real-world applicable. The summary video is available at https://youtu.be/EUVoH-wA-lA.