# Self-Supervised Visual Terrain Classification from Unsupervised Acoustic Feature Learning

arXiv: https://arxiv.org/abs/1912.03227 · HTML: https://mlsanity.com/paper/1912.03227/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2019.

## Built on by (most cited first, 30 of 38)

- [There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge](https://mlsanity.com/paper/2103.01353/) (2021)
- [Fast Traversability Estimation for Wild Visual Navigation](https://mlsanity.com/paper/2305.08510/) (2023)
- [Amodal Panoptic Segmentation](https://mlsanity.com/paper/2202.11542/) (2022)
- [Learning Off-Road Terrain Traversability with Self-Supervisions Only](https://mlsanity.com/paper/2305.18896/) (2023)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)
- [Self-Supervised Traversability Prediction by Learning to Reconstruct Safe Terrain](https://mlsanity.com/paper/2208.01329/) (2022)
- [ScaTE: A Scalable Framework for Self-Supervised Traversability Estimation in Unstructured Environments](https://mlsanity.com/paper/2209.06522/) (2022)
- [EVORA: Deep Evidential Traversability Learning for Risk-Aware Off-Road Autonomy](https://mlsanity.com/paper/2311.06234/) (2023)
- [Wild Visual Navigation: Fast Traversability Learning via Pre-Trained Models and Online Self-Supervision](https://mlsanity.com/paper/2404.07110/) (2024)
- [STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience](https://mlsanity.com/paper/2309.15302/) (2023)
- [RCA: Ride Comfort-Aware Visual Navigation via Self-Supervised Learning](https://mlsanity.com/paper/2207.14460/) (2022)
- [Identifying Terrain Physical Parameters from Vision -- Towards Physical-Parameter-Aware Locomotion and Navigation](https://mlsanity.com/paper/2408.16567/) (2024)
- [Fine-Grained Off-Road Semantic Segmentation and Mapping via Contrastive Learning](https://mlsanity.com/paper/2103.03651/) (2021)
- [Self-Supervised Moving Vehicle Detection from Audio-Visual Cues](https://mlsanity.com/paper/2201.12771/) (2022)
- [Mars Terrain Segmentation with Less Labels](https://mlsanity.com/paper/2202.00791/) (2022)
- [Wait, That Feels Familiar: Learning to Extrapolate Human Preferences for Preference Aligned Path Planning](https://mlsanity.com/paper/2309.09912/) (2023)
- [Contrastive Object Detection Using Knowledge Graph Embeddings](https://mlsanity.com/paper/2112.11366/) (2021)
- [Safe Navigation in Unstructured Environments by Minimizing Uncertainty in Control and Perception](https://mlsanity.com/paper/2306.14601/) (2023)
- [PACER: Preference-conditioned All-terrain Costmap Generation](https://mlsanity.com/paper/2410.23488/) (2024)
- [Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds](https://mlsanity.com/paper/2109.02763/) (2021)
- [AutoGraph: Predicting Lane Graphs from Traffic Observations](https://mlsanity.com/paper/2306.15410/) (2023)
- [Proprioception Is All You Need: Terrain Classification for Boreal Forests](https://mlsanity.com/paper/2403.16877/) (2024)
- [TrackletMapper: Ground Surface Segmentation and Mapping from Traffic Participant Trajectories](https://mlsanity.com/paper/2209.05247/) (2022)
- [An Active and Contrastive Learning Framework for Fine-Grained Off-Road Semantic Segmentation](https://mlsanity.com/paper/2202.09002/) (2022)
- [7th AI Driving Olympics: 1st Place Report for Panoptic Tracking](https://mlsanity.com/paper/2112.05210/) (2021)
- [Improving Deep Dynamics Models for Autonomous Vehicles with Multimodal Latent Mapping of Surfaces](https://mlsanity.com/paper/2303.11756/) (2023)
- [A Multi-modal Fusion Network for Terrain Perception Based on Illumination Aware](https://mlsanity.com/paper/2505.11066/) (2025)
- [Self-Supervised Online Robot-Agnostic Traversability Estimation for Open-World Environments](https://mlsanity.com/paper/2605.28442/) (2026)
- [Progressive Self-Distillation for Ground-to-Aerial Perception Knowledge Transfer](https://mlsanity.com/paper/2208.13404/) (2022)
- [METAVerse: Meta-Learning Traversability Cost Map for Off-Road Navigation](https://mlsanity.com/paper/2307.13991/) (2023)

## Abstract (arXiv)

Mobile robots operating in unknown urban environments encounter a wide range of complex terrains to which they must adapt their planned trajectory for safe and efficient navigation. Most existing approaches utilize supervised learning to classify terrains from either an exteroceptive or a proprioceptive sensor modality. However, this requires a tremendous amount of manual labeling effort for each newly encountered terrain as well as for variations of terrains caused by changing environmental conditions. In this work, we propose a novel terrain classification framework leveraging an unsupervised proprioceptive classifier that learns from vehicle-terrain interaction sounds to self-supervise an exteroceptive classifier for pixel-wise semantic segmentation of images. To this end, we first learn a discriminative embedding space for vehicle-terrain interaction sounds from triplets of audio clips formed using visual features of the corresponding terrain patches and cluster the resulting embeddings. We subsequently use these clusters to label the visual terrain patches by projecting the traversed tracks of the robot into the camera images. Finally, we use the sparsely labeled images to train our semantic segmentation network in a weakly supervised manner. We present extensive quantitative and qualitative results that demonstrate that our proprioceptive terrain classifier exceeds the state-of-the-art among unsupervised methods and our self-supervised exteroceptive semantic segmentation model achieves a comparable performance to supervised learning with manually labeled data.