# Learning to Navigate Using Mid-Level Visual Priors

arXiv: https://arxiv.org/abs/1912.11121 · HTML: https://mlsanity.com/paper/1912.11121/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2019.

## Built on by (most cited first, 25 of 25)

- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning](https://mlsanity.com/paper/2201.10029/) (2022)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation](https://mlsanity.com/paper/2202.02440/) (2022)
- [AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant](https://mlsanity.com/paper/2203.04203/) (2022)
- [CrossLoc: Scalable Aerial Localization Assisted by Multimodal Synthetic Data](https://mlsanity.com/paper/2112.09081/) (2021)
- [GECCO: Geometrically-Conditioned Point Diffusion Models](https://mlsanity.com/paper/2303.05916/) (2023)
- [Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL](https://mlsanity.com/paper/2404.09857/) (2024)
- [Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flight](https://mlsanity.com/paper/2309.09865/) (2023)
- [Environment Predictive Coding for Embodied Agents](https://mlsanity.com/paper/2102.02337/) (2021)
- [Hierarchical Generative Adversarial Imitation Learning with Mid-level Input Generation for Autonomous Driving on Urban Environments](https://mlsanity.com/paper/2302.04823/) (2023)
- [ENTL: Embodied Navigation Trajectory Learner](https://mlsanity.com/paper/2304.02639/) (2023)
- [Online Learning of Reusable Abstract Models for Object Goal Navigation](https://mlsanity.com/paper/2203.02583/) (2022)
- [Object-centric Video Prediction without Annotation](https://mlsanity.com/paper/2105.02799/) (2021)
- [IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation](https://mlsanity.com/paper/2511.17384/) (2025)
- [Invariance is Key to Generalization: Examining the Role of Representation in Sim-to-Real Transfer for Visual Navigation](https://mlsanity.com/paper/2310.15020/) (2023)
- [Using Deep Reinforcement Learning with Automatic Curriculum Learning for Mapless Navigation in Intralogistics](https://mlsanity.com/paper/2202.11512/) (2022)
- [Investigation of Factorized Optical Flows as Mid-Level Representations](https://mlsanity.com/paper/2203.04927/) (2022)
- [Simple Control Baselines for Evaluating Transfer Learning](https://mlsanity.com/paper/2202.03365/) (2022)
- [ASC me to Do Anything: Multi-task Training for Embodied AI](https://mlsanity.com/paper/2202.06987/) (2022)
- [Virtual Guidance as a Mid-level Representation for Navigation with Augmented Reality](https://mlsanity.com/paper/2303.02731/) (2023)
- [Embodied Learning for Lifelong Visual Perception](https://mlsanity.com/paper/2112.14084/) (2021)
- [Object-Centric Action-Enhanced Representations for Robot Visuo-Motor Policy Learning](https://mlsanity.com/paper/2505.20962/) (2025)
- [Stereo4DWalker: Learning 4D-aware Embodied Urban Navigation from Internet Stereo Videos](https://mlsanity.com/paper/2512.10956/) (2025)
- [LASAR: Towards Spatio-temporal Reasoning with Latent Cognitive Map](https://mlsanity.com/paper/2605.16899/) (2026)

## Abstract (arXiv)

How much does having visual priors about the world (e.g. the fact that the world is 3D) assist in learning to perform downstream motor tasks (e.g. navigating a complex environment)? What are the consequences of not utilizing such visual priors in learning? We study these questions by integrating a generic perceptual skill set (a distance estimator, an edge detector, etc.) within a reinforcement learning framework (see Fig. 1). This skill set ("mid-level vision") provides the policy with a more processed state of the world compared to raw images. Our large-scale study demonstrates that using mid-level vision results in policies that learn faster, generalize better, and achieve higher final performance, when compared to learning from scratch and/or using state-of-the-art visual and non-visual representation learning methods. We show that conventional computer vision objectives are particularly effective in this regard and can be conveniently integrated into reinforcement learning frameworks. Finally, we found that no single visual representation was universally useful for all downstream tasks, hence we computationally derive a task-agnostic set of representations optimized to support arbitrary downstream tasks.