mlsanityData to 7 Oct 2026

VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models

arXiv 2404.00210 ↗First posted 30 Mar 2024cs.ROcited by 46

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We propose VLM-Social-Nav, a novel Vision-Language Model (VLM) based navigation approach to compute a robot's motion in human-centered environments. Our goal is to make real-time decisions on robot actions that are socially compliant with human expectations. We utilize a perception model to detect important social entities and prompt a VLM to generate guidance for socially compliant robot behavior. VLM-Social-Nav uses a VLM-based scoring module that computes a cost term that ensures socially appropriate and effective robot actions generated by the underlying planner. Our overall approach reduces reliance on large training datasets and enhances adaptability in decision-making. In practice, it results in improved socially compliant navigation in human-shared environments. We demonstrate and evaluate our system in four different real-world social navigation scenarios with a Turtlebot robot. We observe at least 27.38% improvement in the average success rate and 19.05% improvement in the average collision rate in the four social navigation scenarios. Our user study score shows that VLM-Social-Nav generates the most socially compliant navigation behavior.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 62 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

37 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  2. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  11. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  12. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
23 more
  1. 2022Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigationcited by 122
  2. 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 118
  3. 2020Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehiclescited by 101
  4. 2021Core Challenges of Social Robot Navigation: A Surveycited by 97
  5. 2023SACSoN: Scalable Autonomous Control for Social Navigationcited by 60
  6. 2024Driving Everywhere with Large Language Model Policy Adaptationcited by 53
  7. 2023Principles and Guidelines for Evaluating Social Robot Navigation Algorithmscited by 52
  8. 2020Motion Planning and Control for Mobile Robot Navigation Using Machine Learning: a Surveycited by 48
  9. 2019TH"OR: Human-Robot Navigation Data Collection and Accurate Motion Trajectories Datasetcited by 36
  10. 2023Toward Human-Like Social Robot Navigation: A Large-Scale, Multi-Modal, Social Human Navigation Datasetcited by 29
  11. 2022Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigationcited by 27
  12. 2023How susceptible are LLMs to Logical Fallacies?cited by 23
  13. 2019DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPscited by 21
  14. 2020APPLD: Adaptive Planner Parameter Learning from Demonstrationcited by 17
  15. 2021Conflict Avoidance in Social Navigation -- a Surveycited by 13
  16. 2024DTG : Diffusion-based Trajectory Generation for Mapless Global Navigationcited by 13
  17. 2022A Protocol for Validating Social Navigation Policiescited by 12
  18. 2023MTG: Mapless Trajectory Generator with Traversability Coverage for Outdoor Navigationcited by 10
  19. 2020OF-VO: Efficient Navigation among Pedestrians Using Commodity Sensorscited by 5
  20. 2024VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Trainingcited by 5
  21. 2023A Study on Learning Social Robot Navigation with Multimodal Perceptioncited by 4
  22. 2017Role Playing Learning for Socially Concomitant Mobile Robot Navigationcited by 3
  23. 2023DS-MPEPC: Safe and Deadlock-Avoiding Robot Navigation in Cluttered Dynamic Scenescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 19; 2026: 21.

46 papers.

Sort:

By area:

  1. 2025Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agentscited by 30
  2. 2025A Survey on Efficient Vision-Language Modelscited by 19
  3. 2025A Survey on Large Language Models for Automated Planningcited by 17
  4. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
  5. 2024VL-TGS: Trajectory Generation and Selection using Vision Language Models in Mapless Outdoor Environmentscited by 11
  6. 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigationcited by 10
  7. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 10
  8. 2025LookOut: Real-World Humanoid Egocentric Navigationcited by 9
  9. 2026NavDreamer: Video Models as Zero-Shot 3D Navigatorscited by 9
  10. 2024LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulationcited by 8
  11. 2025Multi-Agent LLM Actor-Critic Framework for Social Robot Navigationcited by 7
  12. 2024Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mindcited by 6
34 more
  1. 2024GSON: A Group-based Social Navigation Framework with Large Multimodal Modelcited by 6
  2. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
  3. 2025Learning to Tune Like an Expert: Interpretable and Scene-Aware Navigation via MLLM Reasoning and CVAE-Based Adaptationcited by 6
  4. 2024Vision Language Models Can Parse Floor Plan Mapscited by 5
  5. 2025ViLAM: Distilling Vision-Language Reasoning into Attention Maps for Social Robot Navigationcited by 5
  6. 2025SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuningcited by 5
  7. 2026From Obstacles to Etiquette: Robot Social Navigation with VLM-Informed Path Selectioncited by 5
  8. 2024InsightSee: Advancing Multi-agent Vision-Language Models for Enhanced Visual Understandingcited by 3
  9. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
  10. 2025ComposableNav: Instruction-Following Navigation in Dynamic Environments via Composable Diffusioncited by 3
  11. 2025EgoCogNav: Cognition-aware Human Egocentric Navigationcited by 3
  12. 2026Contextual Safety Reasoning and Grounding for Open-World Robotscited by 3
  13. 2025LISN: Language-Instructed Social Navigation with VLM-based Controller Modulatingcited by 2
  14. 2025Logic-Guided Socially-aware Robot Navigation World Modelcited by 1
  15. 2026Probing Prompt Design for Socially Compliant Robot Navigation with Vision Language Modelscited by 1
  16. 2026Don't Freeze, Don't Crash: Extending the Safe Operating Range of Neural Navigation in Dense Crowdscited by 1
  17. 2025Perceiving, Reasoning, Adapting: A Dual-Layer Framework for VLM-Guided Precision Robotic Manipulationcited by 0
  18. 2025Trust Through Transparency: Explainable Social Navigation for Autonomous Mobile Robots via Vision-Language Modelscited by 0
  19. 2025CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognitioncited by 0
  20. 2025Tactical Decision for Multi-UGV Confrontation with a Vision-Language Model-Based Commandercited by 0
  21. 2026Human-Aware Robot Behaviour in Self-Driving Labscited by 0
  22. 2026Enhancing Lightweight Vision Language Models through Group Competitive Learning for Socially Compliant Navigationcited by 0
  23. 2026Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectificationcited by 0
  24. 2026Act on What You See: Unlocking Safe Social Navigation in Vision-Language-Action Modelscited by 0
  25. 2026HumAIN: Human-Aware Implicit Social Robot Navigationcited by 0
  26. 2026Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policiescited by 0
  27. 2026The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Modelscited by 0
  28. 2026Visual Grounding in Zero-Shot Vision-Language Controlcited by 0
  29. 2026Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalitiescited by 0
  30. 2026Commonsense-Grounded Path Planning from Abstract Instructionscited by 0
  31. 2026"Dear LLaVA, Please Drive": A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Controlcited by 0
  32. 2026Where Should I Join? Robot Group Joining via Language-Guided Goal Predictioncited by 0
  33. 2026SoGuDiff: Socially Guided Diffusion for Steerable, Norm-Grounded Robot Navigationcited by 0
  34. 2026RECAST: Recasting Vision-Language Semantics into an Actionable Cost Map for Robot Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control
  2. 2026G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation ↗
  3. 2026STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction
  4. 2026SOPD-SocialNav: Selective On-Policy Distillation for Vision-Language Social Navigation
  5. 2024Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces
  6. 2025MAction-SocialNav: Multi-Action Socially Compliant Navigation via Reasoning-enhanced Prompt Tuning
  7. 2021SOCIALGYM: A Framework for Benchmarking Social Robot Navigation
  8. 2024OLiVia-Nav: An Online Lifelong Vision Language Approach for Mobile Robot Social Navigation