Diffusion as Reasoning: Enhancing Object Navigation via Diffusion Model Conditioned on LLM-based Object-Room Knowledge
On the map: LLM reasoning → Spatial and embodied reasoning
also in 1 more
Abstract (arXiv)
The Object Navigation (ObjectNav) task aims to guide an agent to locate target objects in unseen environments using partial observations. Prior approaches have employed location prediction paradigms to achieve long-term goal reasoning, yet these methods often struggle to effectively integrate contextual relation reasoning. Alternatively, map completion-based paradigms predict long-term goals by generating semantic maps of unexplored areas. However, existing methods in this category fail to fully leverage known environmental information, resulting in suboptimal map quality that requires further improvement. In this work, we propose a novel approach to enhancing the ObjectNav task, by training a diffusion model to learn the statistical distribution patterns of objects in semantic maps, and using the map of the explored regions during navigation as the condition to generate the map of the unknown regions, thereby realizing the long-term goal reasoning of the target object, i.e., diffusion as reasoning (DAR). Meanwhile, we propose the Room Guidance method, which leverages commonsense knowledge derived from large language models (LLMs) to guide the diffusion model in generating room-aware object distributions. Based on the generated map in the unknown region, the agent sets the predicted location of the target as the goal and moves towards it. Experiments on Gibson and MP3D show the effectiveness of our method.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 59 references that are in the map. First posted on arXiv 29 Oct 2024.
Builds on
45 of 59 references.
References · 45
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
- 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
- 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
33 more
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
- 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
- 2022Blended Latent Diffusioncited by 430
- 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
- 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
- 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
- 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
- 2020Neural Topological SLAM for Visual Navigationcited by 193
- 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
- 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
- 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equationscited by 172
- 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
- 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 118
- 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
- 2022Navigating to Objects in the Real Worldcited by 104
- 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
- 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
- 2021Auxiliary Tasks and Exploration Enable ObjectNavcited by 73
- 2020Learning Object Relation Graph and Tentative Policy for Visual Navigationcited by 62
- 2023How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontierscited by 53
- 2021VTNet: Visual Transformer Network for Object Goal Navigationcited by 50
- 2023Navigating to Objects Specified by Imagescited by 50
- 2021Learning to Map for Active Semantic Goal Navigationcited by 47
- 2020SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigationcited by 43
- 2021Visual Navigation with Spatial Attentioncited by 43
- 2022Stubborn: A Strong Baseline for Indoor Object Navigationcited by 31
- 2022Zero-shot object goal visual navigationcited by 29
- 2022PEANUT: Predicting and Navigating to Unseen Targetscited by 22
- 2021Agent-Centric Relation Graph for Object Visual Navigationcited by 10
- 2023ENTL: Embodied Navigation Trajectory Learnercited by 9
- 2021Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graphcited by 7
- 2021Map completion from partial observation using the global structure of multiple environmental mapscited by 5
- 2024Leveraging Large Language Model-based Room-Object Relationships Knowledge for Enhancing Multimodal-Input Object Goal Navigationcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Distilling LLM Prior to Flow Model for Generalizable Agent's Imagination in Object Goal Navigation
- 2026Relational Abstractions for Spatial Reasoning with Diffusion Models
- 2026Plug-and-Play Label Map Diffusion for Universal Goal-Oriented Navigation
- 2026From Reactive to Map-Based AI: Tuned Local LLMs for Semantic Zone Inference in Object-Goal Navigation
- 2026HRO: Hierarchical Room-to-Object Framework for Zero-Shot Object Goal Navigation with Large Language Models
- 2024Cognitive Planning for Object Goal Navigation using Generative AI Models
- 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
- 2023Object Goal Navigation with Recursive Implicit Maps