# ViPlanner: Visual Semantic Imperative Learning for Local Navigation

arXiv: https://arxiv.org/abs/2310.00982 · HTML: https://mlsanity.com/paper/2310.00982/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2023.

## Built on by (most cited first, 30 of 46)

- [NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance](https://mlsanity.com/paper/2505.08712/) (2025)
- [TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation](https://mlsanity.com/paper/2505.10696/) (2025)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)
- [NavRL: Learning Safe Flight in Dynamic Environments](https://mlsanity.com/paper/2409.15634/) (2024)
- [NaviDiffusor: Cost-Guided Diffusion Model for Visual Navigation](https://mlsanity.com/paper/2504.10003/) (2025)
- [Prior Does Matter: Visual Navigation via Denoising Diffusion Bridge Models](https://mlsanity.com/paper/2504.10041/) (2025)
- [Grounded Vision-Language Navigation for UAVs with Open-Vocabulary Goal Understanding](https://mlsanity.com/paper/2506.10756/) (2025)
- [SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigation](https://mlsanity.com/paper/2511.21135/) (2025)
- [CREStE: Scalable Mapless Navigation with Internet Scale Priors and Counterfactual Guidance](https://mlsanity.com/paper/2503.03921/) (2025)
- [VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigation](https://mlsanity.com/paper/2510.20818/) (2025)
- [LoGoPlanner: Localization Grounded Navigation Policy with Metric-aware Visual Geometry](https://mlsanity.com/paper/2512.19629/) (2025)
- [Learning Semantic Traversability with Egocentric Video and Automated Annotation Strategy](https://mlsanity.com/paper/2406.02989/) (2024)
- [QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds](https://mlsanity.com/paper/2406.16578/) (2024)
- [Watch Your STEPP: Semantic Traversability Estimation using Pose Projected Features](https://mlsanity.com/paper/2501.17594/) (2025)
- [MetricNet: Recovering Metric Scale in Generative Navigation Policies](https://mlsanity.com/paper/2509.13965/) (2025)
- [MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learning](https://mlsanity.com/paper/2510.03142/) (2025)
- [UrbanVLA: A Vision-Language-Action Model for Urban Micromobility](https://mlsanity.com/paper/2510.23576/) (2025)
- [NavOL: Navigation Policy with Online Imitation Learning](https://mlsanity.com/paper/2605.11762/) (2026)
- [TOP-Nav: Legged Navigation Integrating Terrain, Obstacle and Proprioception Estimation](https://mlsanity.com/paper/2404.15256/) (2024)
- [Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments](https://mlsanity.com/paper/2506.14233/) (2025)
- [VENTURA: Adapting Image Diffusion Models for Unified Task Conditioned Navigation](https://mlsanity.com/paper/2510.01388/) (2025)
- [Towards General Urban Monitoring with Vision-Language Models: A Review, Evaluation, and a Research Agenda](https://mlsanity.com/paper/2510.12400/) (2025)
- [Self-Imitated Diffusion Policy for Efficient and Robust Visual Navigation](https://mlsanity.com/paper/2601.22965/) (2026)
- [FLUX: Accelerating Cross-Embodiment Generative Navigation Policies via Rectified Flow and Static-to-Dynamic Learning](https://mlsanity.com/paper/2603.12806/) (2026)
- [Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies](https://mlsanity.com/paper/2603.12868/) (2026)
- [STRNet: Visual Navigation with Spatio-Temporal Representation through Dynamic Graph Aggregation](https://mlsanity.com/paper/2604.02829/) (2026)
- [Visually-grounded Humanoid Agents](https://mlsanity.com/paper/2604.08509/) (2026)
- [Sight Over Site: Perception-Aware Reinforcement Learning for Efficient Robotic Inspection](https://mlsanity.com/paper/2509.17877/) (2025)
- [Decision-Driven Semantic Object Exploration for Legged Robots via Confidence-Calibrated Perception and Topological Subgoal Selection](https://mlsanity.com/paper/2509.20739/) (2025)
- [ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination](https://mlsanity.com/paper/2512.17435/) (2025)

## Abstract (arXiv)

Real-time path planning in outdoor environments still challenges modern robotic systems due to differences in terrain traversability, diverse obstacles, and the necessity for fast decision-making. Established approaches have primarily focused on geometric navigation solutions, which work well for structured geometric obstacles but have limitations regarding the semantic interpretation of different terrain types and their affordances. Moreover, these methods fail to identify traversable geometric occurrences, such as stairs. To overcome these issues, we introduce ViPlanner, a learned local path planning approach that generates local plans based on geometric and semantic information. The system is trained using the Imperative Learning paradigm, for which the network weights are optimized end-to-end based on the planning task objective. This optimization uses a differentiable formulation of a semantic costmap, which enables the planner to distinguish between the traversability of different terrains and accurately identify obstacles. The semantic information is represented in 30 classes using an RGB colorspace that can effectively encode the multiple levels of traversability. We show that the planner can adapt to diverse real-world environments without requiring any real-world training. In fact, the planner is trained purely in simulation, enabling a highly scalable training data generation. Experimental results demonstrate resistance to noise, zero-shot sim-to-real transfer, and a decrease of 38.02% in terms of traversability cost compared to purely geometric-based approaches. Code and models are made publicly available: https://github.com/leggedrobotics/viplanner.