# Deep Visual MPC-Policy Learning for Navigation

arXiv: https://arxiv.org/abs/1903.02749 · HTML: https://mlsanity.com/paper/1903.02749/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2019.

## Built on by (most cited first, 30 of 56)

- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [ViNT: A Foundation Model for Visual Navigation](https://mlsanity.com/paper/2306.14846/) (2023)
- [GNM: A General Navigation Model to Drive Any Robot](https://mlsanity.com/paper/2210.03370/) (2022)
- [BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments](https://mlsanity.com/paper/2108.03332/) (2021)
- [Large Language Models for Robotics: A Survey](https://mlsanity.com/paper/2311.07226/) (2023)
- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)
- [MaskViT: Masked Visual Pre-Training for Video Prediction](https://mlsanity.com/paper/2206.11894/) (2022)
- [Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning](https://mlsanity.com/paper/2310.10103/) (2023)
- [Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction](https://mlsanity.com/paper/2103.04174/) (2021)
- [SACSoN: Scalable Autonomous Control for Social Navigation](https://mlsanity.com/paper/2306.01874/) (2023)
- [NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance](https://mlsanity.com/paper/2505.08712/) (2025)
- [OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation](https://mlsanity.com/paper/2509.19480/) (2025)
- [VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation](https://mlsanity.com/paper/2407.09829/) (2024)
- [CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models](https://mlsanity.com/paper/2508.13446/) (2025)
- [PanoFlow: Learning 360deg Optical Flow for Surrounding Temporal Understanding](https://mlsanity.com/paper/2202.13388/) (2022)
- [Phone2Proc: Bringing Robust Robots Into Our Chaotic World](https://mlsanity.com/paper/2212.04819/) (2022)
- [Last-Mile Embodied Visual Navigation](https://mlsanity.com/paper/2211.11746/) (2022)
- [NaviDiffusor: Cost-Guided Diffusion Model for Visual Navigation](https://mlsanity.com/paper/2504.10003/) (2025)
- [Prior Does Matter: Visual Navigation via Denoising Diffusion Bridge Models](https://mlsanity.com/paper/2504.10041/) (2025)
- [ExAug: Robot-Conditioned Navigation Policies via Geometric Experience Augmentation](https://mlsanity.com/paper/2210.07450/) (2022)
- [MMVP: Motion-Matrix-based Video Prediction](https://mlsanity.com/paper/2308.16154/) (2023)
- [Learning to Drive Anywhere with Model-Based Reannotation](https://mlsanity.com/paper/2505.05592/) (2025)
- [PlaceNav: Topological Navigation through Place Recognition](https://mlsanity.com/paper/2309.17260/) (2023)
- [AsyncVLA: An Asynchronous VLA for Fast and Robust Navigation on the Edge](https://mlsanity.com/paper/2602.13476/) (2026)
- [HabiCrowd: A High Performance Simulator for Crowd-Aware Visual Navigation](https://mlsanity.com/paper/2306.11377/) (2023)
- [Exploitation-Guided Exploration for Semantic Embodied Navigation](https://mlsanity.com/paper/2311.03357/) (2023)
- [SELFI: Autonomous Self-Improvement with Reinforcement Learning for Social Navigation](https://mlsanity.com/paper/2403.00991/) (2024)
- [NavWAM: A Navigation World Action Model for Goal-Conditioned Visual Navigation](https://mlsanity.com/paper/2606.13494/) (2026)
- [Visual Perspective Taking for Opponent Behavior Modeling](https://mlsanity.com/paper/2105.05145/) (2021)

## Abstract (arXiv)

Humans can routinely follow a trajectory defined by a list of images/landmarks. However, traditional robot navigation methods require accurate mapping of the environment, localization, and planning. Moreover, these methods are sensitive to subtle changes in the environment. In this paper, we propose a Deep Visual MPC-policy learning method that can perform visual navigation while avoiding collisions with unseen objects on the navigation path. Our model PoliNet takes in as input a visual trajectory and the image of the robot's current view and outputs velocity commands for a planning horizon of N steps that optimally balance between trajectory following and obstacle avoidance. PoliNet is trained using a strong image predictive model and traversability estimation model in a MPC setup, with minimal human supervision. Different from prior work, PoliNet can be applied to new scenes without retraining. We show experimentally that the robot can follow a visual trajectory when varying start position and in the presence of previously unseen obstacles. We validated our algorithm with tests both in a realistic simulation environment and in the real world. We also show that we can generate visual trajectories in simulation and execute the corresponding path in the real environment. Our approach outperforms classical approaches as well as previous learning-based baselines in success rate of goal reaching, sub-goal coverage rate, and computational load.