Distilling LLM Prior to Flow Model for Generalizable Agent's Imagination in Object Goal Navigation
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
The Object Goal Navigation (ObjectNav) task challenges agents to locate a specified object in an unseen environment by imagining unobserved regions of the scene. Prior approaches rely on deterministic and discriminative models to complete semantic maps, overlooking the inherent uncertainty in indoor layouts and limiting their ability to generalize to unseen environments. In this work, we propose GOAL, a generative flow-based framework that models the semantic distribution of indoor environments by bridging observed regions with LLM-enriched full-scene semantic maps. During training, spatial priors inferred from large language models (LLMs) are encoded as two-dimensional Gaussian fields and injected into target maps, distilling rich contextual knowledge into the flow model and enabling more generalizable completions. Extensive experiments demonstrate that GOAL achieves state-of-the-art performance on MP3D and Gibson, and shows strong generalization in transfer settings to HM3D. Codes and pretrained models are available at https://github.com/Badi-Li/GOAL.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 71 of its 80 references that are in the map. First posted on arXiv 13 Aug 2025.
Builds on
71 of 80 references.
References · 71
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2017Mask R-CNNcited by 7,259
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2022Scalable Diffusion Models with Transformerscited by 5,931
38 more
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
- 2023Consistency Modelscited by 1,778
- 2021Image Super-Resolution via Iterative Refinementcited by 1,506
- 2020Point Transformercited by 1,462
- 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
- 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
- 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
- 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
- 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
- 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
- 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
- 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
- 2023Improved Techniques for Training Consistency Modelscited by 430
- 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
- 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
- 2024One Step Diffusion via Shortcut Modelscited by 373
- 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
- 2024Navigation World Modelscited by 262
- 2023I2SB: Image-to-Image Schr"odinger Bridgecited by 214
- 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
- 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
- 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
- 2018Visual Semantic Navigation using Scene Priorscited by 127
- 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
- 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 118
- 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
- 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
- 2023Stochastic interpolants with data-dependent couplingscited by 94
- 2018Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learningcited by 91
- 2023Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skillcited by 75
50 most cited of 71 All 71 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Cognitive Planning for Object Goal Navigation using Generative AI Models
- 2026HRO: Hierarchical Room-to-Object Framework for Zero-Shot Object Goal Navigation with Large Language Models
- 2024Diffusion as Reasoning: Enhancing Object Navigation via Diffusion Model Conditioned on LLM-based Object-Room Knowledge
- 2026From Reactive to Map-Based AI: Tuned Local LLMs for Semantic Zone Inference in Object-Goal Navigation
- 2026GoalVLM: VLM-driven Object Goal Navigation for Multi-Agent System
- 2026LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation
- 2025VISTA: Generative Visual Imagination for Vision-and-Language Navigation
- 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning