# Visuospatial Cognitive Assistant

arXiv: https://arxiv.org/abs/2505.12312 · HTML: https://mlsanity.com/paper/2505.12312/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data](https://mlsanity.com/paper/2410.18558/) (2024)
- [GlueStick: Robust Image Matching by Sticking Points and Lines Together](https://mlsanity.com/paper/2304.02008/) (2023)
- [OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text](https://mlsanity.com/paper/2406.08418/) (2024)
- [MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens](https://mlsanity.com/paper/2406.11271/) (2024)

## Built on by (most cited first, 11 of 11)

- [SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models](https://mlsanity.com/paper/2511.23075/) (2025)
- [Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regression](https://mlsanity.com/paper/2511.11239/) (2025)
- [GEM: Generative Supervision Helps Embodied Intelligence](https://mlsanity.com/paper/2605.28548/) (2026)
- [Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning](https://mlsanity.com/paper/2511.16160/) (2025)
- [Towards Predictive, Aligned, and Scalable Robot Learning](https://mlsanity.com/paper/2607.11270/) (2026)
- [Chatting with Images for Introspective Visual Thinking](https://mlsanity.com/paper/2602.11073/) (2026)
- [Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning](https://mlsanity.com/paper/2605.27318/) (2026)
- [Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning](https://mlsanity.com/paper/2606.11719/) (2026)
- [OneCanvas: 3D Scene Understanding via Panoramic Reprojection](https://mlsanity.com/paper/2606.19253/) (2026)
- [SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Experts](https://mlsanity.com/paper/2607.06620/) (2026)
- [Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models](https://mlsanity.com/paper/2608.10278/) (2026)

## Abstract (arXiv)

Video-based spatial cognition is vital for robotics and embodied AI but challenges current Vision-Language Models (VLMs). This paper makes two key contributions. First, we introduce ViCA (Visuospatial Cognitive Assistant)-322K, a diverse dataset of 322,003 QA pairs from real-world indoor videos (ARKitScenes, ScanNet, ScanNet++), offering supervision for 3D metadata-grounded queries and video-based complex reasoning. Second, we develop ViCA-7B, fine-tuned on ViCA-322K, which achieves new state-of-the-art on all eight VSI-Bench tasks, outperforming existing models, including larger ones (e.g., +26.1 on Absolute Distance). For interpretability, we present ViCA-Thinking-2.68K, a dataset with explicit reasoning chains, and fine-tune ViCA-7B to create ViCA-7B-Thinking, a model that articulates its spatial reasoning. Our work highlights the importance of targeted data and suggests paths for improved temporal-spatial modeling. We release all resources to foster research in robust visuospatial intelligence.