mlsanityData to 14 Oct 2026

VideoLLM: Modeling Video Sequence with Large Language Models

arXiv 2305.13292 ↗First posted 22 May 2023cs.CVcited by 116

Not on the map. Why?

Cited by 42 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the exponential growth of video data, there is an urgent need for automated technology to analyze and comprehend video content. However, existing video understanding models are often task-specific and lack a comprehensive capability of handling diverse tasks. The success of large language models (LLMs) like GPT has demonstrated their impressive abilities in sequence causal reasoning. Building upon this insight, we propose a novel framework called VideoLLM that leverages the sequence reasoning capabilities of pre-trained LLMs from natural language processing (NLP) for video sequence understanding. VideoLLM incorporates a carefully designed Modality Encoder and Semantic Translator, which convert inputs from various modalities into a unified token sequence. This token sequence is then fed into a decoder-only LLM. Subsequently, with the aid of a simple task head, our VideoLLM yields an effective unified framework for different kinds of video understanding tasks. To evaluate the efficacy of VideoLLM, we conduct extensive experiments using multiple LLMs and fine-tuning methods. We evaluate our VideoLLM on eight tasks sourced from four different datasets. The experimental results demonstrate that the understanding and reasoning capabilities of LLMs can be effectively transferred to video understanding tasks. We release the code at https://github.com/cg1177/VideoLLM.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Builds on 91 of its 103 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

91 of 103 references.

References · 91

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  12. 2020End-to-End Object Detection with Transformerscited by 7,414
38 more
  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  13. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  14. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  15. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  16. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  17. 2017The Kinetics Human Action Video Datasetcited by 1,909
  18. 2018SlowFast Networks for Video Recognitioncited by 1,867
  19. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  20. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  21. 2021ViViT: A Video Vision Transformercited by 1,620
  22. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  23. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  24. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  25. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  26. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  27. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  28. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  29. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  30. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  31. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  32. 2021Multiscale Vision Transformerscited by 897
  33. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  34. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  35. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  36. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  37. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  38. 2020Rescaling Egocentric Visioncited by 540

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 28; 2024: 37; 2025: 39; 2026: 12.

116 papers.

Sort:

By area:

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  5. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  6. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  7. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  8. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  9. 2023Video Understanding with Large Language Models: A Surveycited by 206
  10. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  11. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  12. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 125
38 more
  1. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  2. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  3. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 108
  4. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  5. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
  6. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  7. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 81
  8. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  9. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 75
  10. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  11. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  12. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  13. 2024Needle In A Multimodal Haystackcited by 58
  14. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 56
  15. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  16. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 49
  17. 2025SnapMoGen: Human Motion Generation from Expressive Textscited by 47
  18. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  19. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  20. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  21. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  22. 2024Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Modelscited by 33
  23. 2025AToken: A Unified Tokenizer for Visioncited by 30
  24. 2023PALM: Predicting Actions through Language Modelscited by 29
  25. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  26. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 24
  27. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
  28. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  29. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Modelscited by 19
  30. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  31. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19
  32. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understandingcited by 18
  33. 2024Causal Evaluation of Language Modelscited by 18
  34. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18
  35. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  36. 2025Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Modelcited by 17
  37. 2025HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understandingcited by 17
  38. 2025Advancing vision-language models in front-end development via data synthesiscited by 16

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  2. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models
  3. 2023VTimeLLM: Empower LLM to Grasp Video Moments
  4. 2026Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understanding
  5. 2025SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding
  6. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens
  7. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  8. 2024Do Language Models Understand Time?