# Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models arXiv: https://arxiv.org/abs/2409.15451 ยท HTML: https://mlsanity.com/paper/2409.15451/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. - Builds on 45 of its 54 references that are in the map. - First posted on arXiv 23 Sep 2024. ## Placed in - LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/) ## Builds on (most cited first, 30 of 45) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017) - [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021) - [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022) - [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022) - [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022) - [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018) - [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021) - [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021) - [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020) - [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022) - [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023) - [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023) - [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023) - [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023) - [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019) - [ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects](https://mlsanity.com/paper/2006.13171/) (2020) - [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024) - [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://mlsanity.com/paper/2203.10421/) (2022) - [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019) - [Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization](https://mlsanity.com/paper/2201.13360/) (2022) - [ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings](https://mlsanity.com/paper/2206.12403/) (2022) - [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022) - [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020) - [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016) - [ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation](https://mlsanity.com/paper/2301.13166/) (2023) - [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022) - [MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects](https://mlsanity.com/paper/1804.09194/) (2018) ## Built on by (most cited first, 12 of 12) - [Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answering](https://mlsanity.com/paper/2507.12846/) (2025) - [ForesightNav: Learning Scene Imagination for Efficient Exploration](https://mlsanity.com/paper/2504.16062/) (2025) - [SpatialLLM: From Multi-modality Data to Urban Spatial Intelligence](https://mlsanity.com/paper/2505.12703/) (2025) - [IPPON: Common Sense Guided Informative Path Planning for Object Goal Navigation](https://mlsanity.com/paper/2410.19697/) (2024) - [FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning](https://mlsanity.com/paper/2604.03696/) (2026) - [GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering](https://mlsanity.com/paper/2506.01174/) (2025) - [osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning](https://mlsanity.com/paper/2507.12753/) (2025) - [WildOS: Open-Vocabulary Object Search in the Wild](https://mlsanity.com/paper/2602.19308/) (2026) - [CRAFT: Grounded Multi-Agent Coordination Under Partial Information](https://mlsanity.com/paper/2603.25268/) (2026) - [Efficient Navigation in Unknown Indoor Environments with Vision-Language Models](https://mlsanity.com/paper/2510.04991/) (2025) - [360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents](https://mlsanity.com/paper/2608.08814/) (2026) - [A Scene Language Model for Open-Vocabulary Scene Mapping](https://mlsanity.com/paper/2609.21400/) (2026) ## Abstract (arXiv) Large Language Models (LLM) have emerged as a tool for robots to generate task plans using common sense reasoning. For the LLM to generate actionable plans, scene context must be provided, often through a map. Recent works have shifted from explicit maps with fixed semantic classes to implicit open vocabulary maps based on queryable embeddings capable of representing any semantic class. However, embeddings cannot directly report the scene context as they are implicit, requiring further processing for LLM integration. To address this, we propose an explicit text-based map that can represent thousands of semantic classes while easily integrating with LLMs due to their text-based nature by building upon large-scale image recognition models. We study how entities in our map can be localized and show through evaluations that our text-based map localizations perform comparably to those from open vocabulary maps while using two to four orders of magnitude less memory. Real-robot experiments demonstrate the grounding of an LLM with the text-based map to solve user tasks.