# STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience

arXiv: https://arxiv.org/abs/2309.15302 · HTML: https://mlsanity.com/paper/2309.15302/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 30 of 31)

- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [BADGR: An Autonomous Self-Supervised Learning-Based Navigation System](https://mlsanity.com/paper/2002.05700/) (2020)
- [Motion Planning and Control for Mobile Robot Navigation Using Machine Learning: a Survey](https://mlsanity.com/paper/2011.13112/) (2020)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [Fast Traversability Estimation for Wild Visual Navigation](https://mlsanity.com/paper/2305.08510/) (2023)
- [Self-Supervised Visual Terrain Classification from Unsupervised Acoustic Feature Learning](https://mlsanity.com/paper/1912.03227/) (2019)
- [Safe Robot Navigation via Multi-Modal Anomaly Detection](https://mlsanity.com/paper/2001.07934/) (2020)
- [VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigation](https://mlsanity.com/paper/2105.09371/) (2021)
- [TerrainNet: Visual Modeling of Complex Terrain for High-speed, Off-road Navigation](https://mlsanity.com/paper/2303.15771/) (2023)
- [LaND: Learning to Navigate from Disengagements](https://mlsanity.com/paper/2010.04689/) (2020)
- [GrASPE: Graph based Multimodal Fusion for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2209.05722/) (2022)
- [Learning Visual Locomotion with Cross-Modal Supervision](https://mlsanity.com/paper/2211.03785/) (2022)
- [VI-IKD: High-Speed Accurate Off-Road Navigation using Learned Visual-Inertial Inverse Kinodynamics](https://mlsanity.com/paper/2203.15983/) (2022)
- [Toward Wheeled Mobility on Vertically Challenging Terrain: Platforms, Datasets, and Algorithms](https://mlsanity.com/paper/2303.00998/) (2023)
- [Reinforced Grounded Action Transformation for Sim-to-Real Transfer](https://mlsanity.com/paper/2008.01279/) (2020)
- [Visual Representation Learning for Preference-Aware Path Planning](https://mlsanity.com/paper/2109.08968/) (2021)
- [Stochastic Grounded Action Transformation for Robot Learning in Simulation](https://mlsanity.com/paper/2008.01281/) (2020)
- [Learning Risk-Aware Costmaps via Inverse Reinforcement Learning for Off-Road Navigation](https://mlsanity.com/paper/2302.00134/) (2023)
- [High-Speed Accurate Robot Control using Learned Forward Kinodynamics and Non-linear Least Squares Optimization](https://mlsanity.com/paper/2206.08487/) (2022)
- [Learning-on-the-Drive: Self-supervised Adaptation of Visual Offroad Traversability Models](https://mlsanity.com/paper/2306.15226/) (2023)
- [Adversarial Imitation Learning from Video using a State Observer](https://mlsanity.com/paper/2202.00243/) (2022)
- [RCA: Ride Comfort-Aware Visual Navigation via Self-Supervised Learning](https://mlsanity.com/paper/2207.14460/) (2022)
- [Wait, That Feels Familiar: Learning to Extrapolate Human Preferences for Preference Aligned Path Planning](https://mlsanity.com/paper/2309.09912/) (2023)
- [Autonomous Visual Assistance for Robot Operations Using a Tethered UAV](https://mlsanity.com/paper/1904.00078/) (2019)

## Built on by (most cited first, 11 of 11)

- [EgoPet: Egomotion and Interaction Data from an Animal's Perspective](https://mlsanity.com/paper/2404.09991/) (2024)
- [CREStE: Scalable Mapless Navigation with Internet Scale Priors and Counterfactual Guidance](https://mlsanity.com/paper/2503.03921/) (2025)
- [SLR: Learning Quadruped Locomotion without Privileged Information](https://mlsanity.com/paper/2406.04835/) (2024)
- [VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Training](https://mlsanity.com/paper/2403.08109/) (2024)
- [Learning Semantic Traversability with Egocentric Video and Automated Annotation Strategy](https://mlsanity.com/paper/2406.02989/) (2024)
- [PACER: Preference-conditioned All-terrain Costmap Generation](https://mlsanity.com/paper/2410.23488/) (2024)
- [TOP-Nav: Legged Navigation Integrating Terrain, Obstacle and Proprioception Estimation](https://mlsanity.com/paper/2404.15256/) (2024)
- [Autonomous UAV Flight Navigation in Confined Spaces: A Reinforcement Learning Approach](https://mlsanity.com/paper/2508.16807/) (2025)
- [Self-Supervised Online Robot-Agnostic Traversability Estimation for Open-World Environments](https://mlsanity.com/paper/2605.28442/) (2026)
- [Lift, Splat, Map: Lifting Foundation Masks for Label-Free Semantic Scene Completion](https://mlsanity.com/paper/2407.03425/) (2024)
- [Aligning Robot Navigation Behaviors with Human Intentions and Preferences](https://mlsanity.com/paper/2409.18982/) (2024)

## Abstract (arXiv)

Terrain awareness, i.e., the ability to identify and distinguish different types of terrain, is a critical ability that robots must have to succeed at autonomous off-road navigation. Current approaches that provide robots with this awareness either rely on labeled data which is expensive to collect, engineered features and cost functions that may not generalize, or expert human demonstrations which may not be available. Towards endowing robots with terrain awareness without these limitations, we introduce Self-supervised TErrain Representation LearnING (STERLING), a novel approach for learning terrain representations that relies solely on easy-to-collect, unconstrained (e.g., non-expert), and unlabelled robot experience, with no additional constraints on data collection. STERLING employs a novel multi-modal self-supervision objective through non-contrastive representation learning to learn relevant terrain representations for terrain-aware navigation. Through physical robot experiments in off-road environments, we evaluate STERLING features on the task of preference-aligned visual navigation and find that STERLING features perform on par with fully supervised approaches and outperform other state-of-the-art methods with respect to preference alignment. Additionally, we perform a large-scale experiment of autonomously hiking a 3-mile long trail which STERLING completes successfully with only two manual interventions, demonstrating its robustness to real-world off-road conditions.