CoCoSI: Collaborative Cognitive Map Construction for Spatial Intelligence
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Spatial intelligence is a key frontier for multimodal large language models (MLLMs), enabling them to reason about the physical world from visual experience. Inspired by human spatial cognition, recent approaches construct grid-based cognitive maps from multi-frame visual inputs to maintain coherent spatial representations over time. However, limited context lengths still challenge spatial understanding, while existing methods, such as long-context modeling and external memory, often require architectural changes, memory modules, or finetuning, limiting their applicability to off-the-shelf pretrained MLLMs. This motivates a lightweight, model-agnostic method for preserving spatial information beyond the native context window. To this end, we propose a plug-and-play multi-agent framework that collaboratively constructs cognitive maps as structured spatial memory, enhancing the spatial understanding of arbitrary pretrained MLLMs without architectural modification or additional training. Our framework features local-global agent coordination, cognitive map construction with atomic commits, and cross-agent verification. Extensive experiments demonstrate that our method achieves superior performance on spatial understanding tasks while remaining fully training-free. Code will be released.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 32 references that are in the map. First posted on arXiv 9 Jun 2026.
Builds on
30 of 32 references.
Closest ancestors · 3
Same direction, newest first
- 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence29 May
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
Other references · 27
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
- 2025OpenAI GPT-5 System Cardcited by 910
- 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
- 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
15 more
- 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
- 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
- 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
- 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
- 2020Learning to Explore using Active Neural SLAMcited by 308
- 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
- 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
- 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
- 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
- 2017Neural Map: Structured Memory for Deep Reinforcement Learningcited by 88
- 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
- 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
- 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
- 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 34
- 2025LongVideoAgent: Multi-Agent Reasoning with Long Videoscited by 23
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
- 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition
- 2026SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks
- 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science
- 2025Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning
- 2026Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models
- 2025From reactive to cognitive: brain-inspired spatial intelligence for embodied agents
- 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes