mlsanityData to 7 Oct 2026

ST-LLM: Large Language Models Are Effective Temporal Learners

arXiv 2404.00308 ↗First posted 30 Mar 2024cs.CVcited by 126

Not on the map. Why?

Cited by 58 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have showcased impressive capabilities in text comprehension and generation, prompting research efforts towards video LLMs to facilitate human-AI interaction at the video level. However, how to effectively encode and understand videos in video-based dialogue systems remains to be solved. In this paper, we investigate a straightforward yet unexplored question: Can we feed all spatial-temporal tokens into the LLM, thus delegating the task of video sequence modeling to the LLMs? Surprisingly, this simple approach yields significant improvements in video understanding. Based upon this, we propose ST-LLM, an effective video-LLM baseline with Spatial-Temporal sequence modeling inside LLM. Furthermore, to address the overhead and stability issues introduced by uncompressed video tokens within LLMs, we develop a dynamic masking strategy with tailor-made training objectives. For particularly long videos, we have also designed a global-local input module to balance efficiency and effectiveness. Consequently, we harness LLM for proficient spatial-temporal modeling, while upholding efficiency and stability. Extensive experimental results attest to the effectiveness of our method. Through a more concise model and training pipeline, ST-LLM establishes a new state-of-the-art result on VideoChatGPT-Bench and MVBench. Codes have been available at https://github.com/TencentARC/ST-LLM.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 47 of its 54 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

47 of 54 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
35 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  13. 2023PaLM 2 Technical Reportcited by 1,154
  14. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  15. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  17. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  19. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  20. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  21. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  22. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  23. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  24. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  25. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  26. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  27. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  28. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  29. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  30. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  31. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  32. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  33. 2023Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferringcited by 57
  34. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41
  35. 2016Deep Learning for Video Classification and Captioningcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 46; 2025: 60; 2026: 20.

126 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  3. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  4. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  5. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  6. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  7. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  8. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  9. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  10. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  11. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  12. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
38 more
  1. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  2. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  3. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  4. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 80
  5. 2024Vript: A Video Is Worth Thousands of Wordscited by 71
  6. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  7. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  8. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  9. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  10. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  11. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 44
  12. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  13. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 40
  14. 2024Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMscited by 39
  15. 2024Autoregressive Models in Vision: A Surveycited by 39
  16. 2024Lost in Time: A New Temporal Benchmark for VideoLLMscited by 36
  17. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 35
  18. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  19. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  20. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  21. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  22. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  23. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  24. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  25. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 25
  26. 2025TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understandingcited by 23
  27. 2024STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Trainingcited by 22
  28. 2025MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representationscited by 22
  29. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  30. 2024Towards Event-oriented Long Video Understandingcited by 20
  31. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  32. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  33. 2025Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Modelscited by 19
  34. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18
  35. 2025Memory-enhanced Retrieval Augmentation for Long Video Understandingcited by 17
  36. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  37. 2025Vid-SME: Membership Inference Attacks against Large Video Understanding Modelscited by 17
  38. 2024VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentationcited by 14

50 most cited of 126 All 126 (JSON)