mlsanityData to 14 Oct 2026

Koala: Key frame-conditioned long video-LLM

arXiv 2404.04346 ↗First posted 5 Apr 2024cs.CVcited by 57

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships. State-of-the-art video Large Language Models (vLLMs) hold promise as a viable solution due to their demonstrated emergent capabilities on new tasks. However, despite being trained on millions of short seconds-long videos, vLLMs are unable to understand minutes-long videos and accurately answer questions about them. To address this limitation, we propose a lightweight and self-supervised approach, Key frame-conditioned long video-LLM (Koala), that introduces learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to longer videos. Our approach introduces two new tokenizers that condition on visual tokens computed from sparse video key frames for understanding short and long video moments. We train our proposed approach on HowTo100M and demonstrate its effectiveness on zero-shot long video understanding benchmarks, where it outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across all tasks. Surprisingly, we also empirically show that our approach not only helps a pretrained vLLM to understand long videos but also improves its accuracy on short-term action recognition.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 64 of its 82 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

64 of 82 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  3. 2018SlowFast Networks for Video Recognitioncited by 1,867
  4. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  5. 2021ViViT: A Video Vision Transformercited by 1,620
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  8. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  14. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  15. 2021Multiscale Vision Transformerscited by 897
  16. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  17. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  18. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  19. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  20. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  21. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  22. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  23. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  24. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  25. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  26. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  27. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  28. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  29. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  30. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  31. 2021Video Transformer Networkcited by 272
  32. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  33. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  34. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  35. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  36. 2021Anticipative Video Transformercited by 200
  37. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  38. 2021Towards Long-Form Video Understandingcited by 146

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 39; 2026: 4.

57 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  2. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  3. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  4. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  5. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  6. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 114
  7. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  8. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  9. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 58
  10. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 54
  11. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videoscited by 44
  12. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 43
38 more
  1. 2024Temporal Reasoning Transfer from Text to Videocited by 38
  2. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 34
  3. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  4. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 30
  5. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  6. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  7. 2025Seek-CAD: A Self-refined Generative Modeling for 3D Parametric CAD Using Local Inference via DeepSeekcited by 24
  8. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 23
  9. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 22
  10. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Videocited by 21
  11. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 19
  12. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processingcited by 18
  13. 2024SEAL: Semantic Attention Learning for Long Video Representationcited by 13
  14. 2025K-frames: Scene-Driven Any-k Keyframe Selection for long video understandingcited by 11
  15. 2024Enhancing Temporal Modeling of Video LLMs via Time Gatingcited by 10
  16. 2025Map the Flow: Revealing Hidden Pathways of Information in VideoLLMscited by 10
  17. 2025Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMscited by 9
  18. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  19. 2025AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understandingcited by 9
  20. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  21. 2024AdaCM2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reductioncited by 7
  22. 2024VideoSAVi: Self-Aligned Video Language Models without Human Supervisioncited by 7
  23. 2025PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assemblycited by 7
  24. 2024FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filteringcited by 5
  25. 2025VITED: Video Temporal Evidence Distillationcited by 5
  26. 2025Protecting Your Video Content: Disrupting Automated Video-based LLM Annotationscited by 5
  27. 2025CogStream: Context-guided Streaming Video Question Answeringcited by 5
  28. 2025Language-Guided Temporal Token Pruning for Efficient VideoLLM Processingcited by 5
  29. 2024Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experiencecited by 4
  30. 2025LeAdQA: LLM-Driven Context-Aware Temporal Grounding for Video Question Answeringcited by 4
  31. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLMcited by 4
  32. 2024Extending Video Masked Autoencoders to 128 framescited by 3
  33. 2024FLAASH: Flow-Attention Adaptive Semantic Hierarchical Fusion for Multi-Modal Tobacco Content Analysiscited by 2
  34. 2025REEF: Relevance-Aware and Efficient LLM Adapter for Video Understandingcited by 2
  35. 2025TriPSS: A Tri-Modal Keyframe Extraction Framework Using Perceptual, Structural, and Semantic Representationscited by 2
  36. 2026TiFRe: Text-guided Video Frame Reduction for Efficient Video Multi-modal Large Language Modelscited by 2
  37. 2025A Review on Large Language Models for Visual Analyticscited by 1
  38. 2025ReSpec: Relevance and Specificity Grounded Online Filtering for Learning on Video-Text Data Streamscited by 1

50 most cited of 57 All 57 (JSON)