mlsanityData to 14 Oct 2026

PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning

arXiv 2404.16994 ↗First posted 25 Apr 2024cs.CVcited by 322

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Vision-language pre-training has significantly elevated performance across a wide range of image-language applications. Yet, the pre-training process for video-related tasks demands exceptionally large computational and data resources, which hinders the progress of video-language models. This paper investigates a straight-forward, highly efficient, and resource-light approach to adapting an existing image-language pre-trained model for dense video understanding. Our preliminary experiments reveal that directly fine-tuning pre-trained image-language models with multiple frames as inputs on video datasets leads to performance saturation or even a drop. Our further investigation reveals that it is largely attributed to the bias of learned high-norm visual features. Motivated by this finding, we propose a simple but effective pooling strategy to smooth the feature distribution along the temporal dimension and thus reduce the dominant impacts from the extreme features. The new model is termed Pooling LLaVA, or PLLaVA in short. PLLaVA achieves new state-of-the-art performance on modern benchmark datasets for both video question-answer and captioning tasks. Notably, on the recent popular VideoChatGPT benchmark, PLLaVA achieves a score of 3.48 out of 5 on average of five evaluated dimensions, exceeding the previous SOTA results from GPT4V (IG-VLM) by 9%. On the latest multi-choice benchmark MVBench, PLLaVA achieves 58.1% accuracy on average across 20 sub-tasks, 14.5% higher than GPT4V (IG-VLM). Code is available at https://pllava.github.io/

Key facts. Cited by 322 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 50 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

44 of 50 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024LITA: Language Instructed Temporal-Localization Assistant27 Mar
  2. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios7 Mar
  3. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
26 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  2. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  3. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  6. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  7. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  8. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  9. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  11. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  12. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  13. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  14. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  15. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  16. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  17. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  18. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  19. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  20. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  21. 2023Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokenscited by 71
  22. 2021AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsamplingcited by 47
  23. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41
  24. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
  25. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  26. 2023A Large Cross-Modal Video Retrieval Dataset with Reading Comprehensioncited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 99; 2025: 154; 2026: 69.

322 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  4. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  5. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  6. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  7. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  8. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  9. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
  10. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  11. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  12. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
38 more
  1. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  2. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  3. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  4. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  5. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  6. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  7. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  8. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 167
  9. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 164
  10. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  11. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  12. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  13. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  14. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  15. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  16. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128
  17. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  18. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  19. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  20. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  21. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 100
  22. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 96
  23. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
  24. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
  25. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  26. 2025Scaling RL to Long Videoscited by 84
  27. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
  28. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  29. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 75
  30. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  31. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 70
  32. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 70
  33. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 70
  34. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 68
  35. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 65
  36. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 65
  37. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 61
  38. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 60

50 most cited of 322 All 322 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Frames2LoRA: Parametric Video Internalization for Vision-Language Models
  2. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
  3. 2025VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning
  4. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models
  5. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  6. 2025Q-Adapter: Visual Query Adapter for Extracting Textually-related Features in Video Captioning
  7. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  8. 2024Distilling Vision-Language Models on Millions of Videos