MLFM: Multi-Layered Feature Maps for Richer Language Understanding in Zero-Shot Semantic Navigation
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Recent progress in large vision-language models has driven improvements in language-based semantic navigation, where an embodied agent must reach a target object described in natural language. Yet we still lack a clear, language-focused evaluation framework to test how well agents ground the words in their instructions. We address this gap by proposing LangNav, an open-vocabulary multi-object navigation dataset with natural language goal descriptions (e.g. 'go to the red short candle on the table') and corresponding fine-grained linguistic annotations (e.g., attributes: color=red, size=short; relations: support=on). These labels enable systematic evaluation of language understanding. To evaluate on this setting, we extend multi-object navigation task setting to Language-guided Multi-Object Navigation (LaMoN), where the agent must find a sequence of goals specified using language. Furthermore, we propose Multi-Layered Feature Map (MLFM), a novel method that builds a queryable, multi-layered semantic map from pretrained vision-language features and proves effective for reasoning over fine-grained attributes and spatial relations in goal descriptions. Experiments on LangNav show that MLFM outperforms state-of-the-art zero-shot mapping-based navigation baselines.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 44 references that are in the map. First posted on arXiv 9 Jul 2025.
Builds on
38 of 44 references.
Closest ancestors · 2
Same direction, newest first
- 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation6 Dec
- 2022Visual Language Maps for Robot Navigation11 Oct
Other references · 36
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
- 2022Language-driven Semantic Segmentationcited by 711
- 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
- 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
24 more
- 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
- 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
- 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
- 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
- 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
- 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
- 2022Navigating to Objects in the Real Worldcited by 104
- 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
- 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
- 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
- 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
- 2023GOAT: GO to Any Thingcited by 89
- 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
- 2025MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigationcited by 55
- 2022Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instancescited by 53
- 2022Retrospectives on the Embodied AI Workshopcited by 46
- 20223D-Aware Object Goal Navigation via Simultaneous Exploration and Identificationcited by 46
- 2025SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesiscited by 26
- 2023Object Goal Navigation with Recursive Implicit Mapscited by 16
- 2025Eval3D: Interpretable and Fine-grained Evaluation for 3D Generationcited by 15
- 2024One Map to Find Them All: Real-time Open-Vocabulary Mapping for Zero-shot Multi-Object Navigationcited by 13
- 2023MOPA: Modular Object Navigation with PointGoal Agentscited by 11
- 2025ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Groundingcited by 7
- 2024Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigationcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Instance-Enriched Semantic Maps for Visual Language Navigation
- 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
- 2025OpenNav: Open-World Navigation with Multimodal Large Language Models
- 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains
- 2026Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
- 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
- 2025A Navigation Framework Utilizing Vision-Language Models
- 2026VISOR: VIsual Spatial Object Reasoning for Language-driven Object Navigation