mlsanityData to 7 Oct 2026

Seeing is Believing (and Predicting): Context-Aware Multi-Human Behavior Prediction with Vision Language Models

arXiv 2512.15957 ↗First posted 17 Dec 2025cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Accurately predicting human behaviors is crucial for mobile robots operating in human-populated environments. While prior research primarily focuses on predicting actions in single-human scenarios from an egocentric view, several robotic applications require understanding multiple human behaviors from a third-person perspective. To this end, we present CAMP-VLM (Context-Aware Multi-human behavior Prediction): a Vision Language Model (VLM)-based framework that incorporates contextual features from visual input and spatial awareness from scene graphs to enhance prediction of humans-scene interactions. Due to the lack of suitable datasets for multi-human behavior prediction from an observer view, we perform fine-tuning of CAMP-VLM with synthetic human behavior data generated by a photorealistic simulator, and evaluate the resulting models on both synthetic and real-world sequences to assess their generalization capabilities. Leveraging Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), CAMP-VLM outperforms the best-performing baseline by up to 66.9% in prediction accuracy.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 51 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

36 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insights1 Apr

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  8. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  10. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  11. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  12. 2018VirtualHome: Simulating Household Activities via Programscited by 411
23 more
  1. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  2. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  3. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  4. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  5. 2019Human Motion Trajectory Prediction: A Surveycited by 231
  6. 2018Human Action Recognition and Prediction: A Surveycited by 226
  7. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  8. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 106
  9. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  10. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 79
  11. 2020In the Eye of the Beholder: Gaze and Actions in First Person Videocited by 68
  12. 2020LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activitiescited by 46
  13. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  14. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  15. 2023PALM: Predicting Actions through Language Modelscited by 28
  16. 2019Relational Action Forecastingcited by 27
  17. 2024An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Modelscited by 22
  18. 2024Long-Term Human Trajectory Prediction using 3D Dynamic Scene Graphscited by 15
  19. 2023GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planningcited by 13
  20. 2023A Survey on Deep Learning Techniques for Action Anticipationcited by 11
  21. 2024Human Action Anticipation: A Surveycited by 8
  22. 2024On the Efficacy of Text-Based Input Modalities for Action Anticipationcited by 6
  23. 2024HiMemFormer: Hierarchical Memory-Aware Transformer for Multi-Agent Action Anticipationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026HINT-Plan: Human Intention-Aware Robot Task Planning in Context-Rich Environments using Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context
  2. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions
  3. 2025VLM-driven Behavior Tree for Context-aware Task Planning
  4. 2026Lightweight Visual Reasoning for Socially-Aware Robots
  5. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  6. 2025A Survey on Efficient Vision-Language-Action Models
  7. 2026Wall-OSS-0.5 Technical Report
  8. 2026Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning