mlsanityData to 14 Oct 2026

The Potential and Limitations of Vision-Language Models for Human Motion Understanding: A Case Study in Data-Driven Stroke Rehabilitation

arXiv 2511.17727 ↗First posted 21 Nov 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models (VLMs) have demonstrated remarkable performance across a wide range of computer-vision tasks, sparking interest in their potential for digital health applications. Here, we apply VLMs to two fundamental challenges in data-driven stroke rehabilitation: automatic quantification of rehabilitation dose and impairment from videos. We formulate these problems as motion-identification tasks, which can be addressed using VLMs. We evaluate our proposed framework on a cohort of 29 healthy controls and 51 stroke survivors. Our results show that current VLMs lack the fine-grained motion understanding required for precise quantification: dose estimates are comparable to a baseline that excludes visual information, and impairment scores cannot be reliably predicted. Nevertheless, several findings suggest future promise. With optimized prompting and post-processing, VLMs can classify high-level activities from a few frames, detect motion and grasp with moderate accuracy, and approximate dose counts within 25% of ground truth for mildly impaired and healthy participants, all without task-specific training or finetuning. These results highlight both the current limitations and emerging opportunities of VLMs for data-driven stroke rehabilitation and broader clinical video analysis.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 58 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

26 of 58 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  9. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  11. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  12. 2024Long Context Transfer from Language to Visioncited by 515
14 more
  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  2. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
  3. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  4. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 234
  5. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 87
  6. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 73
  7. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  8. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 33
  9. 2024Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mindcited by 28
  10. 2024ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognitioncited by 21
  11. 2025Video Action Differencingcited by 14
  12. 2020Towards data-driven stroke rehabilitation via wearable sensors and deep learningcited by 3
  13. 2021PrimSeq: a deep learning-based pipeline to quantitate rehabilitation trainingcited by 2
  14. 2023Quantifying Impairment and Disease Severity Using AI Models Trained on Healthy Subjectscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Decoding Children's Gait Behaviorcited by 1
  2. 2026Agentic AI for Personalized Physiotherapy: A Multi-Agent Framework for Generative Video Training and Real-Time Pose Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Can Vision Language Models Judge Action Quality? An Empirical Evaluation
  2. 2025Reducing Annotation Burden in Physical Activity Research Using Vision-Language Models
  3. 2024Enhancing Gait Video Analysis in Neurodegenerative Diseases by Knowledge Augmentation in Vision Language Model
  4. 2024@Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology
  5. 2025Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings
  6. 2026Vision-Language Models for Occupational Physical Exposure Assessment: Estimating External Hand Forces in Manual Material Handling Tasks from RGB Video
  7. 2025Wearable-informed generative digital avatars predict task-conditioned post-stroke locomotion
  8. 2025Visual Language Models show widespread visual deficits on neuropsychological tests