mlsanityData to 7 Oct 2026

Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input

arXiv 2408.15542 ↗First posted 28 Aug 2024cs.CV cs.AI cs.MMcited by 132

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Rapid advancements have been made in extending Large Language Models (LLMs) to Large Multi-modal Models (LMMs). However, extending input modality of LLMs to video data remains a challenging endeavor, especially for long videos. Due to insufficient access to large-scale high-quality video data and the excessive compression of visual features, current methods exhibit limitations in effectively processing long videos. In this paper, we introduce Kangaroo, a powerful Video LMM aimed at addressing these challenges. Confronted with issue of inadequate training data, we develop a data curation system to build a large-scale dataset with high-quality annotations for vision-language pre-training and instruction tuning. In addition, we design a curriculum training pipeline with gradually increasing resolution and number of input frames to accommodate long videos. Evaluation results demonstrate that, with 8B parameters, Kangaroo achieves state-of-the-art performance across a variety of video understanding benchmarks while exhibiting competitive results on others. Particularly, on benchmarks specialized for long videos, Kangaroo excels some larger models with over 10B parameters and proprietary models.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 78 of its 97 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

78 of 97 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos26 Aug
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  3. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output3 Jul
  4. 2024Long Context Transfer from Language to Vision24 Jun
  5. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding20 Jun
  6. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoning17 Jun
  7. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  9. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  10. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  11. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
3 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  2. 2023Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks7 Jun
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
38 more
  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  2. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  11. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  13. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  14. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  15. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  16. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  17. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  18. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  19. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  20. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  21. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  22. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  23. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  24. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  25. 2024InternLM2 Technical Reportcited by 391
  26. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  27. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  28. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  29. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  30. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  31. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  32. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  33. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  34. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  35. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  36. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  37. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  38. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 76; 2026: 31.

132 papers.

Sort:

By area:

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  4. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  5. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  6. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  7. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  8. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  9. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  10. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  11. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  12. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
38 more
  1. 2025Scaling RL to Long Videoscited by 83
  2. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  3. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  4. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
  5. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 72
  6. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  7. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  8. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  9. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  10. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  11. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  12. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  13. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  14. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  15. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 46
  16. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  17. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 46
  18. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  19. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  20. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 38
  21. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 36
  22. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  23. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  24. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  25. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 33
  26. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 33
  27. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  28. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  29. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  30. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  31. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  32. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  33. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  34. 2025Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understandingcited by 28
  35. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  36. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 26
  37. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  38. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 25

50 most cited of 132 All 132 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  2. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
  3. 2024Understanding Long Videos with Multimodal Language Models
  4. 2025Video Panels for Long Video Understanding
  5. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding
  6. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
  7. 2024World Model on Million-Length Video And Language With Blockwise RingAttention
  8. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos