MUSON: A Reasoning-oriented Multimodal Dataset for Socially Compliant Navigation in Urban Environments
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
Socially compliant navigation requires structured reasoning about dynamic pedestrians and physical constraints to ensure safe and interpretable decisions. Vision-language models (VLMs) provide a promising foundation for this task because they can integrate visual observations with language-based social knowledge. However, existing untuned VLMs still struggle to reliably understand fine-grained social norms, making task-specific fine-tuning essential. At the same time, no large-scale egocentric dataset is available this task. To address these challenges, we introduce MUSON, a multimodal dataset for short-horizon social navigation containing 10,110 egocentric samples collected across diverse indoor and outdoor social scenes. MUSON adopts a structured five-step chain-of-thought annotation framework comprising perception, prediction, reasoning, action, and explanation. It explicitly models static physical constraints and employs a standardized six-action decision space. Compared with existing social-navigation datasets, MUSON provides consistent annotations for reasoning, actions, and explanations. We evaluate ten representative small-to-medium VLMs on MUSON. Qwen3-VL-8B achieves the strongest decision-level performance, attaining the highest action accuracy of 0.7765 and Macro-F1 score of 0.7490, as well as the lowest collision rate of 0.0609. These results demonstrate that MUSON is an effective and reusable benchmark for advancing socially compliant navigation. The dataset is publicly available at https://github.com/MUSON-dataset/MUSON/releases/tag/v1.0.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 39 references that are in the map. First posted on arXiv 28 Dec 2025.
Builds on
34 of 39 references.
Closest ancestors · 2
Same direction, newest first
- 2024Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces30 Dec
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul
Other references · 32
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Qwen Technical Reportcited by 3,273
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2024TinyLlama: An Open-Source Small Language Modelcited by 604
20 more
- 2017End-to-end Driving via Conditional Imitation Learningcited by 479
- 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
- 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
- 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
- 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
- 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
- 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
- 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
- 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
- 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
- 2022Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigationcited by 122
- 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
- 2023ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Applicationcited by 48
- 2024CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environmentscited by 43
- 2023Toward Human-Like Social Robot Navigation: A Large-Scale, Multi-Modal, Social Human Navigation Datasetcited by 29
- 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigationcited by 10
- 2024GSON: A Group-based Social Navigation Framework with Large Multimodal Modelcited by 6
- 2025SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuningcited by 5
- 2025CostNav: A Navigation Benchmark for Real-World Economic-Cost Evaluation of Physical AI Agentscited by 1
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Probing Prompt Design for Socially Compliant Robot Navigation with Vision Language Modelscited by 1
- 2026Enhancing Lightweight Vision Language Models through Group Competitive Learning for Socially Compliant Navigationcited by 0
- 2026Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Controlcited by 0
- 2026Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policiescited by 0
- 2026Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies13 Jul 2026
- 2026Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control27 Jun 2026
- 2026Enhancing Lightweight Vision Language Models through Group Competitive Learning for Socially Compliant Navigation12 Mar 2026
- 2026Probing Prompt Design for Socially Compliant Robot Navigation with Vision Language Models21 Jan 2026
Similar, not cited
Close in wording, no citation either way
- 2026STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction
- 2025MAction-SocialNav: Multi-Action Socially Compliant Navigation via Reasoning-enhanced Prompt Tuning
- 2025AutoSpatial: Visual-Language Reasoning for Social Robot Navigation through Efficient Spatial Reasoning Learning
- 2026Act on What You See: Unlocking Safe Social Navigation in Vision-Language-Action Models
- 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks
- 2025ViLAM: Distilling Vision-Language Reasoning into Attention Maps for Social Robot Navigation
- 2026SOPD-SocialNav: Selective On-Policy Distillation for Vision-Language Social Navigation
- 2026From Obstacles to Etiquette: Robot Social Navigation with VLM-Informed Path Selection