mlsanityData to 7 Oct 2026

Can Local Vision-Language Models improve Activity Recognition over Vision Transformers? -- Case Study on Newborn Resuscitation

arXiv 2602.12002 ↗First posted 12 Feb 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Accurate documentation of newborn resuscitation is essential for quality improvement and adherence to clinical guidelines, yet remains underutilized in practice. Previous work using 3D-CNNs and Vision Transformers (ViT) has shown promising results in detecting key activities from newborn resuscitation videos, but also highlighted the challenges in recognizing such fine-grained activities. This work investigates the potential of generative AI (GenAI) methods to improve activity recognition from such videos. Specifically, we explore the use of local vision-language models (VLMs), combined with large language models (LLMs), and compare them to a supervised TimeSFormer baseline. Using a simulated dataset comprising 13.26 hours of newborn resuscitation videos, we evaluate several zero-shot VLM-based strategies and fine-tuned VLMs with classification heads, including Low-Rank Adaptation (LoRA). Our results suggest that small (local) VLMs struggle with hallucinations, but when fine-tuned with LoRA, the results reach F1 score at 0.91, surpassing the TimeSformer results of 0.70.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 19 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

9 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review4 Mar

Other references · 8

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023Mistral 7Bcited by 3,065
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  7. 2022SVFormer: Semi-supervised Video Transformer for Action Recognitioncited by 67
  8. 2023Activity Recognition From Newborn Resuscitation Videoscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings
  2. 2024Vision-language models for decoding provider attention during neonatal resuscitation
  3. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  4. 2023Medical Vision Language Pretraining: A survey
  5. 2025Vision Language Models in Medicine
  6. 2025The Potential and Limitations of Vision-Language Models for Human Motion Understanding: A Case Study in Data-Driven Stroke Rehabilitation
  7. 2025Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks
  8. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models