# NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistants

arXiv: https://arxiv.org/abs/2502.13894 · HTML: https://mlsanity.com/paper/2502.13894/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 56 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Builds on (most cited first, 30 of 51)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [Generating Images with Multimodal Language Models](https://mlsanity.com/paper/2305.17216/) (2023)
- [ViNT: A Foundation Model for Visual Navigation](https://mlsanity.com/paper/2306.14846/) (2023)
- [Neural Topological SLAM for Visual Navigation](https://mlsanity.com/paper/2005.12256/) (2020)
- [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://mlsanity.com/paper/2203.10421/) (2022)
- [Guiding Instruction-based Image Editing via Multimodal Large Language Models](https://mlsanity.com/paper/2309.17102/) (2023)
- [ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings](https://mlsanity.com/paper/2206.12403/) (2022)
- [InstructDiffusion: A Generalist Modeling Interface for Vision Tasks](https://mlsanity.com/paper/2309.03895/) (2023)
- [ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2312.16217/) (2023)
- [PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning](https://mlsanity.com/paper/2201.10029/) (2022)
- [GameFactory: Creating New Games with Generative Interactive Videos](https://mlsanity.com/paper/2501.08325/) (2025)
- [ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation](https://mlsanity.com/paper/2301.13166/) (2023)
- [L3MVN: Leveraging Large Language Models for Visual Target Navigation](https://mlsanity.com/paper/2304.05501/) (2023)
- [Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale](https://mlsanity.com/paper/2204.03514/) (2022)
- [Compositional Foundation Models for Hierarchical Planning](https://mlsanity.com/paper/2309.08587/) (2023)

## Built on by (most cited first, 11 of 11)

- [RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2506.04308/) (2025)
- [TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2510.07181/) (2025)
- [A Survey of Interactive Generative Video](https://mlsanity.com/paper/2504.21853/) (2025)
- [VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning](https://mlsanity.com/paper/2506.09049/) (2025)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation](https://mlsanity.com/paper/2512.14691/) (2025)
- [CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusion](https://mlsanity.com/paper/2506.14769/) (2025)
- [Hierarchical Scoring with 3D Gaussian Splatting for Instance Image-Goal Navigation](https://mlsanity.com/paper/2506.07338/) (2025)
- [GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies](https://mlsanity.com/paper/2511.00998/) (2025)
- [Think before Go: Hierarchical Reasoning for Image-goal Navigation](https://mlsanity.com/paper/2604.17407/) (2026)
- [Schr"odinger's Navigator: Imagining an Ensemble of Futures for Zero-Shot Object Navigation](https://mlsanity.com/paper/2512.21201/) (2025)

## Abstract (arXiv)

Navigating unfamiliar environments presents significant challenges for household robots, requiring the ability to recognize and reason about novel decoration and layout. Existing reinforcement learning methods cannot be directly transferred to new environments, as they typically rely on extensive mapping and exploration, leading to time-consuming and inefficient. To address these challenges, we try to transfer the logical knowledge and the generalization ability of pre-trained foundation models to zero-shot navigation. By integrating a large vision-language model with a diffusion network, our approach named mname ~constructs a visual predictor that continuously predicts the agent's potential observations in the next step which can assist robots generate robust actions. Furthermore, to adapt the temporal property of navigation, we introduce temporal historical information to ensure that the predicted image is aligned with the navigation scene. We then carefully designed an information fusion framework that embeds the predicted future frames as guidance into goal-reaching policy to solve downstream image navigation tasks. This approach enhances navigation control and generalization across both simulated and real-world environments. Through extensive experimentation, we demonstrate the robustness and versatility of our method, showcasing its potential to improve the efficiency and effectiveness of robotic navigation in diverse settings.