# Visual Representation Learning for Preference-Aware Path Planning

arXiv: https://arxiv.org/abs/2109.08968 · HTML: https://mlsanity.com/paper/2109.08968/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 18 Sep 2021.

## Builds on (most cited first, 12 of 12)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [BADGR: An Autonomous Self-Supervised Learning-Based Navigation System](https://mlsanity.com/paper/2002.05700/) (2020)
- [Visual Representations for Semantic Target Driven Navigation](https://mlsanity.com/paper/1805.06066/) (2018)
- [HarDNet: A Low Memory Traffic Network](https://mlsanity.com/paper/1909.00948/) (2019)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)
- [GONet: A Semi-Supervised Deep Learning Approach For Traversability Estimation](https://mlsanity.com/paper/1803.03254/) (2018)
- [Situational Fusion of Visual Representation for Visual Navigation](https://mlsanity.com/paper/1908.09073/) (2019)
- [The STRANDS Project: Long-Term Autonomy in Everyday Environments](https://mlsanity.com/paper/1604.04384/) (2016)
- [Learning Deployable Navigation Policies at Kilometer Scale from a Single Traversal](https://mlsanity.com/paper/1807.05211/) (2018)

## Built on by (most cited first, 18 of 18)

- [Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation](https://mlsanity.com/paper/2209.10780/) (2022)
- [DTG : Diffusion-based Trajectory Generation for Mapless Global Navigation](https://mlsanity.com/paper/2403.09900/) (2024)
- [High-Speed Accurate Robot Control using Learned Forward Kinodynamics and Non-linear Least Squares Optimization](https://mlsanity.com/paper/2206.08487/) (2022)
- [STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience](https://mlsanity.com/paper/2309.15302/) (2023)
- [CREStE: Scalable Mapless Navigation with Internet Scale Priors and Counterfactual Guidance](https://mlsanity.com/paper/2503.03921/) (2025)
- [PIETRA: Physics-Informed Evidential Learning for Traversing Out-of-Distribution Terrain](https://mlsanity.com/paper/2409.03005/) (2024)
- [Wait, That Feels Familiar: Learning to Extrapolate Human Preferences for Preference Aligned Path Planning](https://mlsanity.com/paper/2309.09912/) (2023)
- [Reinforcement Learning for Wheeled Mobility on Vertically Challenging Terrain](https://mlsanity.com/paper/2409.02383/) (2024)
- [VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Training](https://mlsanity.com/paper/2403.08109/) (2024)
- [Learning Semantic Traversability with Egocentric Video and Automated Annotation Strategy](https://mlsanity.com/paper/2406.02989/) (2024)
- [PACER: Preference-conditioned All-terrain Costmap Generation](https://mlsanity.com/paper/2410.23488/) (2024)
- [A Study on Learning Social Robot Navigation with Multimodal Perception](https://mlsanity.com/paper/2309.12568/) (2023)
- [OVerSeeC: Open-Vocabulary Costmap Generation from Satellite Images and Natural Language](https://mlsanity.com/paper/2602.18606/) (2026)
- [Context-Conditional Navigation with a Learning-Based Terrain- and Robot-Aware Dynamics Model](https://mlsanity.com/paper/2307.09206/) (2023)
- [Dyna-LfLH: Learning Agile Navigation in Dynamic Environments from Learned Hallucination](https://mlsanity.com/paper/2403.17231/) (2024)
- [Learning a Terrain- and Robot-Aware Dynamics Model for Autonomous Mobile Robot Navigation](https://mlsanity.com/paper/2409.11452/) (2024)
- [Lift, Splat, Map: Lifting Foundation Masks for Label-Free Semantic Scene Completion](https://mlsanity.com/paper/2407.03425/) (2024)
- [Aligning Robot Navigation Behaviors with Human Intentions and Preferences](https://mlsanity.com/paper/2409.18982/) (2024)

## Abstract (arXiv)

Autonomous mobile robots deployed in outdoor environments must reason about different types of terrain for both safety (e.g., prefer dirt over mud) and deployer preferences (e.g., prefer dirt path over flower beds). Most existing solutions to this preference-aware path planning problem use semantic segmentation to classify terrain types from camera images, and then ascribe costs to each type. Unfortunately, there are three key limitations of such approaches -- they 1) require pre-enumeration of the discrete terrain types, 2) are unable to handle hybrid terrain types (e.g., grassy dirt), and 3) require expensive labelled data to train visual semantic segmentation. We introduce Visual Representation Learning for Preference-Aware Path Planning (VRL-PAP), an alternative approach that overcomes all three limitations: VRL-PAP leverages unlabeled human demonstrations of navigation to autonomously generate triplets for learning visual representations of terrain that are viewpoint invariant and encode terrain types in a continuous representation space. The learned representations are then used along with the same unlabeled human navigation demonstrations to learn a mapping from the representation space to terrain costs. At run time, VRL-PAP maps from images to representations and then representations to costs to perform preference-aware path planning. We present empirical results from challenging outdoor settings that demonstrate VRL-PAP 1) is successfully able to pick paths that reflect demonstrated preferences, 2) is comparable in execution to geometric navigation with a highly detailed manually annotated map (without requiring such annotations), 3) is able to generalize to novel terrain types with minimal additional unlabeled demonstrations.