# VT-LVLM-AR: A Video-Temporal Large Vision-Language Model Adapter for Fine-Grained Action Recognition in Long-Term Videos

arXiv: https://arxiv.org/abs/2508.15903 · HTML: https://mlsanity.com/paper/2508.15903/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 19 of 19)

- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models](https://mlsanity.com/paper/2305.15023/) (2023)
- [A Comprehensive Study of Deep Video Action Recognition](https://mlsanity.com/paper/2012.06567/) (2020)
- [Long-term Temporal Convolutions for Action Recognition](https://mlsanity.com/paper/1604.04494/) (2016)
- [Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback](https://mlsanity.com/paper/2501.01377/) (2025)
- [StNet: Local and Global Spatial-Temporal Modeling for Action Recognition](https://mlsanity.com/paper/1811.01549/) (2018)
- [LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning](https://mlsanity.com/paper/2503.15621/) (2025)
- [A Survey on Efficient Vision-Language Models](https://mlsanity.com/paper/2504.09724/) (2025)
- [PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models](https://mlsanity.com/paper/2504.14117/) (2025)
- [AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability](https://mlsanity.com/paper/2405.14129/) (2024)
- [2D versus 3D Convolutional Spiking Neural Networks Trained with Unsupervised STDP for Human Action Recognition](https://mlsanity.com/paper/2205.13474/) (2022)
- [End-to-end Video-level Representation Learning for Action Recognition](https://mlsanity.com/paper/1711.04161/) (2017)

## Built on by (most cited first, 1 of 1)

- [Resource-Efficient RGB-Only Action Recognition for Edge Deployment](https://mlsanity.com/paper/2602.10818/) (2026)

## Abstract (arXiv)

Human action recognition in long-term videos, characterized by complex backgrounds and subtle action differences, poses significant challenges for traditional deep learning models due to computational overhead, difficulty in capturing long-range temporal dependencies, and limited semantic understanding. While Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have shown remarkable capabilities in multi-modal understanding and reasoning, their direct application to continuous video streams for fine-grained action recognition remains an open problem. This paper introduces VT-LVLM-AR (Video-Temporal Large Vision-Language Model Adapter for Action Recognition), a novel framework designed to bridge this gap. VT-LVLM-AR comprises a Video-to-Event Mapper (VTEM) that efficiently transforms raw video into compact, semantically rich, and temporally coherent "visual event sequences" through lightweight spatio-temporal feature extraction, adaptive temporal pooling, and conceptual quantization with an event coherence bias. These visual event sequences are then fed into an LVLM-based Action Reasoning module, specifically a frozen LLaVA-1.5 model, adapted using parameter-efficient Prompt Tuning (P-Tuning v2) for action classification. Comprehensive evaluations on the NTU RGB+D and NTU RGB+D 120 datasets demonstrate that VT-LVLM-AR consistently achieves state-of-the-art performance, surpassing existing methods (e.g., 94.1% accuracy on NTU RGB+D X-Sub). Ablation studies confirm the critical contributions of VTEM's components and the efficacy of Prompt Tuning, while human evaluations underscore the interpretability of our visual event representations. This work highlights the immense potential of leveraging LVLMs for robust and interpretable video action understanding through effective video-to-language translation and efficient model adaptation.