ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Visual navigation is a fundamental capability for autonomous home-assistance robots, enabling long-horizon tasks such as object search. While recent methods have leveraged Large Language Models (LLMs) to incorporate commonsense reasoning and improve exploration efficiency, their planning remains constrained by textual representations, which cannot adequately capture spatial occupancy or scene geometry--critical factors for navigation decisions. We explore whether Vision-Language Models (VLMs) can achieve mapless visual navigation using only onboard RGB/RGB-D streams, unlocking their potential for spatial perception and planning. We achieve this through an imagination-powered navigation framework, ImagineNav++, which imagines future observation images from candidate robot views and translates navigation planning into a simple best-view image selection problem for VLMs. First, a future-view imagination module distills human navigation preferences to generate semantically meaningful viewpoints with high exploration potential. These imagined views then serve as visual prompts for the VLM to identify the most informative viewpoint. To maintain spatial consistency, we develop a selective foveation memory mechanism, which hierarchically integrates keyframe observations via a sparse-to-dense framework, constructing a compact yet comprehensive memory for long-term spatial reasoning. This approach transforms goal-oriented navigation into a series of tractable point-goal navigation tasks. Extensive experiments on open-vocabulary object and instance navigation benchmarks show that ImagineNav++ achieves SOTA performance in mapless settings, even surpassing most map-based methods, highlighting the importance of scene imagination and memory in VLM-based spatial reasoning.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 89 of its 101 references that are in the map. First posted on arXiv 19 Dec 2025.
Builds on
89 of 101 references.
Closest ancestors · 8
Same direction, newest first
- 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigation20 Feb
- 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering8 Nov
- 2024ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination13 Oct
- 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models16 Feb
- 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
- 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation6 Dec
- 2023Multimodal Large Language Model for Visual Navigation12 Oct
- 2022Visual Language Maps for Robot Navigation11 Oct
Other references · 81
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Segment Anythingcited by 8,234
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2023Visual Instruction Tuningcited by 7,869
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2017Mask R-CNNcited by 7,277
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
- 2023VILA: On Pre-training for Visual Language Modelscited by 692
- 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
- 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
- 2018On Evaluation of Embodied Navigation Agentscited by 547
- 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
- 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
- 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
- 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
- 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
- 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
- 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
- 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
- 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
- 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
- 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 167
- 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 156
- 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 150
- 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 127
- 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 124
- 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 120
- 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
- 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
- 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
- 2023ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Imagecited by 111
- 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 109
- 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 106
- 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 99
50 most cited of 81 All 81 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026MVP-Nav: Multi-layer Value Map Planner Navigatorcited by 1
- 2026MVP-Nav: Multi-layer Value Map Planner Navigator30 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2025VISTA: Generative Visual Imagination for Vision-and-Language Navigation
- 2025A Navigation Framework Utilizing Vision-Language Models
- 2026ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics
- 2026FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-Language Navigation
- 2025VISTAv2: World Imagination for Indoor Vision-and-Language Navigation
- 2026GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
- 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models
- 2026AgentVLN: Towards Agentic Vision-and-Language Navigation