# NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance

arXiv: https://arxiv.org/abs/2505.08712 · HTML: https://mlsanity.com/paper/2505.08712/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 51 of its 57 references that are in the map.
- First posted on arXiv 13 May 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 51)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering](https://mlsanity.com/paper/2312.00109/) (2023)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Gaussian Splatting SLAM](https://mlsanity.com/paper/2312.06741/) (2023)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration](https://mlsanity.com/paper/2310.07896/) (2023)
- [NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation](https://mlsanity.com/paper/2402.15852/) (2024)
- [ViNT: A Foundation Model for Visual Navigation](https://mlsanity.com/paper/2306.14846/) (2023)
- [NaVILA: Legged Robot Vision-Language-Action Model for Navigation](https://mlsanity.com/paper/2412.04453/) (2024)
- [VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation](https://mlsanity.com/paper/2312.03275/) (2023)
- [GNM: A General Navigation Model to Drive Any Robot](https://mlsanity.com/paper/2210.03370/) (2022)
- [Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation](https://mlsanity.com/paper/2306.11290/) (2023)
- [Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation](https://mlsanity.com/paper/2405.07503/) (2024)
- [DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation](https://mlsanity.com/paper/2403.07788/) (2024)
- [Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks](https://mlsanity.com/paper/2412.06224/) (2024)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)

## Built on by (most cited first, 30 of 49)

- [Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparities](https://mlsanity.com/paper/2507.13019/) (2025)
- [InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layouts](https://mlsanity.com/paper/2509.10813/) (2025)
- [VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation](https://mlsanity.com/paper/2512.19021/) (2025)
- [DreamNav: A Trajectory-Based Imaginative Framework for Zero-Shot Vision-and-Language Navigation](https://mlsanity.com/paper/2509.11197/) (2025)
- [TextOp: Real-time Interactive Text-Driven Humanoid Robot Motion Generation and Control](https://mlsanity.com/paper/2602.07439/) (2026)
- [TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments](https://mlsanity.com/paper/2602.02459/) (2026)
- [LoGoPlanner: Localization Grounded Navigation Policy with Metric-aware Visual Geometry](https://mlsanity.com/paper/2512.19629/) (2025)
- [textscNaVIDA: Vision-Language Navigation with Inverse Dynamics Augmentation](https://mlsanity.com/paper/2601.18188/) (2026)
- [MetricNet: Recovering Metric Scale in Generative Navigation Policies](https://mlsanity.com/paper/2509.13965/) (2025)
- [MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learning](https://mlsanity.com/paper/2510.03142/) (2025)
- [UrbanVLA: A Vision-Language-Action Model for Urban Micromobility](https://mlsanity.com/paper/2510.23576/) (2025)
- [MG-Nav: Dual-Scale Visual Navigation via Sparse Spatial Memory](https://mlsanity.com/paper/2511.22609/) (2025)
- [SynthVerse: A Large-Scale Diverse Synthetic Dataset for Point Tracking](https://mlsanity.com/paper/2602.04441/) (2026)
- [NavOL: Navigation Policy with Online Imitation Learning](https://mlsanity.com/paper/2605.11762/) (2026)
- [Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation](https://mlsanity.com/paper/2605.27582/) (2026)
- [HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness](https://mlsanity.com/paper/2609.15195/) (2026)
- [DiffVL: Diffusion-Based Visual Localization on 2D Maps via BEV-Conditioned GPS Denoising](https://mlsanity.com/paper/2509.14565/) (2025)
- [CostNav: A Navigation Benchmark for Real-World Economic-Cost Evaluation of Physical AI Agents](https://mlsanity.com/paper/2511.20216/) (2025)
- [A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Control](https://mlsanity.com/paper/2601.06133/) (2026)
- [Self-Imitated Diffusion Policy for Efficient and Robust Visual Navigation](https://mlsanity.com/paper/2601.22965/) (2026)
- [FLUX: Accelerating Cross-Embodiment Generative Navigation Policies via Rectified Flow and Static-to-Dynamic Learning](https://mlsanity.com/paper/2603.12806/) (2026)
- [Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies](https://mlsanity.com/paper/2603.12868/) (2026)
- [UniNav: A Unified World-Action Diffusion Model for Visual Navigation](https://mlsanity.com/paper/2608.03244/) (2026)
- [Sight Over Site: Perception-Aware Reinforcement Learning for Efficient Robotic Inspection](https://mlsanity.com/paper/2509.17877/) (2025)
- [Deep Active Inference with Diffusion Policy and Multiple Timescale World Model for Real-World Exploration and Navigation](https://mlsanity.com/paper/2510.23258/) (2025)
- [What Limits Vision-and-Language Navigation ?](https://mlsanity.com/paper/2605.13328/) (2026)
- [X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching](https://mlsanity.com/paper/2607.28560/) (2026)
- [Steering Generative Robot Policies with Lexicographic Preferences](https://mlsanity.com/paper/2609.15014/) (2026)
- [ISS Policy : Scalable Diffusion Policy with Implicit Scene Supervision](https://mlsanity.com/paper/2512.15020/) (2025)
- [LabBuilder: Protocol-Grounded 3D Layout Generation for Interactable and Safe Laboratory](https://mlsanity.com/paper/2605.02288/) (2026)

## Abstract (arXiv)

Learning to navigate in dynamic and complex open-world environments is a critical yet challenging capability for autonomous robots. Existing approaches often rely on cascaded modular frameworks, which require extensive hyperparameter tuning or learning from limited real-world demonstration data. In this paper, we propose Navigation Diffusion Policy (NavDP), an end-to-end network trained solely in simulation that enables zero-shot sim-to-real transfer across diverse environments and robot embodiments. The core of NavDP is a unified transformer-based architecture that jointly learns trajectory generation and trajectory evaluation, both conditioned solely on local RGB-D observation. By learning to predict critic values for contrastive trajectory samples, our proposed approach effectively leverages supervision from privileged information available in simulation, thereby fostering accurate spatial understanding and enabling the distinction between safe and dangerous behaviors. To support this, we develop an efficient data generation pipeline in simulation and construct a large-scale dataset encompassing over one million meters of navigation experience across 3,000 scenes. Empirical experiments in both simulated and real-world environments demonstrate that NavDP significantly outperforms prior state-of-the-art methods. Furthermore, we identify key factors influencing the generalization performance of NavDP. The dataset and code are publicly available at https://wzcai99.github.io/navigation-diffusion-policy.github.io.