mlsanityData to 7 Oct 2026

Diffusion as Reasoning: Enhancing Object Navigation via Diffusion Model Conditioned on LLM-based Object-Room Knowledge

arXiv 2410.21842 ↗First posted 29 Oct 2024cs.CV cs.AIcited by 1

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

The Object Navigation (ObjectNav) task aims to guide an agent to locate target objects in unseen environments using partial observations. Prior approaches have employed location prediction paradigms to achieve long-term goal reasoning, yet these methods often struggle to effectively integrate contextual relation reasoning. Alternatively, map completion-based paradigms predict long-term goals by generating semantic maps of unexplored areas. However, existing methods in this category fail to fully leverage known environmental information, resulting in suboptimal map quality that requires further improvement. In this work, we propose a novel approach to enhancing the ObjectNav task, by training a diffusion model to learn the statistical distribution patterns of objects in semantic maps, and using the map of the explored regions during navigation as the condition to generate the map of the unknown regions, thereby realizing the long-term goal reasoning of the target object, i.e., diffusion as reasoning (DAR). Meanwhile, we propose the Room Guidance method, which leverages commonsense knowledge derived from large language models (LLMs) to guide the diffusion model in generating room-aware object distributions. Based on the generated map in the unknown region, the agent sets the predicted location of the target as the goal and moves towards it. Experiments on Gibson and MP3D show the effectiveness of our method.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 59 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

45 of 59 references.

References · 45

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  10. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  11. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
33 more
  1. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  2. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  3. 2022Blended Latent Diffusioncited by 430
  4. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  5. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  6. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  7. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  8. 2020Neural Topological SLAM for Visual Navigationcited by 193
  9. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  10. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  11. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equationscited by 172
  12. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  13. 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 118
  14. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  15. 2022Navigating to Objects in the Real Worldcited by 104
  16. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  17. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  18. 2021Auxiliary Tasks and Exploration Enable ObjectNavcited by 73
  19. 2020Learning Object Relation Graph and Tentative Policy for Visual Navigationcited by 62
  20. 2023How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontierscited by 53
  21. 2021VTNet: Visual Transformer Network for Object Goal Navigationcited by 50
  22. 2023Navigating to Objects Specified by Imagescited by 50
  23. 2021Learning to Map for Active Semantic Goal Navigationcited by 47
  24. 2020SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigationcited by 43
  25. 2021Visual Navigation with Spatial Attentioncited by 43
  26. 2022Stubborn: A Strong Baseline for Indoor Object Navigationcited by 31
  27. 2022Zero-shot object goal visual navigationcited by 29
  28. 2022PEANUT: Predicting and Navigating to Unseen Targetscited by 22
  29. 2021Agent-Centric Relation Graph for Object Visual Navigationcited by 10
  30. 2023ENTL: Embodied Navigation Trajectory Learnercited by 9
  31. 2021Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graphcited by 7
  32. 2021Map completion from partial observation using the global structure of multiple environmental mapscited by 5
  33. 2024Leveraging Large Language Model-based Room-Object Relationships Knowledge for Enhancing Multimodal-Input Object Goal Navigationcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025Distilling LLM Prior to Flow Model for Generalizable Agent's Imagination in Object Goal Navigation
  2. 2026Relational Abstractions for Spatial Reasoning with Diffusion Models
  3. 2026Plug-and-Play Label Map Diffusion for Universal Goal-Oriented Navigation
  4. 2026From Reactive to Map-Based AI: Tuned Local LLMs for Semantic Zone Inference in Object-Goal Navigation
  5. 2026HRO: Hierarchical Room-to-Object Framework for Zero-Shot Object Goal Navigation with Large Language Models
  6. 2024Cognitive Planning for Object Goal Navigation using Generative AI Models
  7. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
  8. 2023Object Goal Navigation with Recursive Implicit Maps