mlsanityData to 14 Oct 2026

VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling

arXiv 2501.00574 ↗First posted 31 Dec 2024cs.CV cs.LGcited by 220

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Long-context video modeling is critical for multimodal large language models (MLLMs), enabling them to process movies, online video streams, and so on. Despite its advances, handling long videos remains challenging due to the difficulty in efficiently understanding the extremely long video context. This paper aims to address this issue from aspects of model architecture, training data, training strategy and evaluation benchmark. First, we propose a novel Hierarchical video token Compression (HiCo) method, which leverages visual redundancy in long videos to compress long video context from Clip-level to Video-level, reducing the computation significantly while preserving essential details, achieving an extreme compression ratio of approximately 1/50 with almost no performance loss. Second, we introduce a multi-stage short-to-long learning scheme, a large-scale dataset of real-world long videos named LongVid, and a challenging ``Multi-Hop Needle-In-A-Video-Haystack'' benchmark. Finally, we build a powerful video MLLM named VideoChat-Flash, which shows a leading performance on both mainstream long and short video benchmarks at the 2B and 7B model scale. It first gets 99.1% accuracy over 10,000 frames in NIAH among open-source models.

Key facts. Cited by 220 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 70 of its 84 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

70 of 84 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning25 Oct
  2. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct
  3. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding22 Sep
  4. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture4 Sep
  5. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  6. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos26 Aug
  7. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  8. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  9. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output3 Jul
  10. 2024Long Context Transfer from Language to Vision24 Jun
  11. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding20 Jun
  12. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoning17 Jun
9 more
  1. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding13 Jun
  2. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  4. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  5. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  6. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 49

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
37 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2022Token Merging: Your ViT But Fastercited by 1,021
  4. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  7. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  9. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  10. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  11. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  12. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  13. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  14. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  15. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  16. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  17. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 285
  18. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
  19. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  20. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  21. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  22. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 190
  23. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  24. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  25. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  26. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 114
  27. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  28. 2021Negative Sample Matters: A Renaissance of Metric Learning for Temporal Groundingcited by 90
  29. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 89
  30. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 86
  31. 2022LVOS: A Benchmark for Long-term Video Object Segmentationcited by 77
  32. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  33. 2024Vript: A Video Is Worth Thousands of Wordscited by 71
  34. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  35. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  36. 2023LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answeringcited by 25
  37. 2024COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Trainingcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 103; 2026: 117.

220 papers.

Sort:

By area:

  1. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
  2. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  3. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
  4. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 139
  5. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  6. 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 134
  7. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98
  8. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  9. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 79
  10. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  11. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65
  12. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 64
38 more
  1. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  2. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  3. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 53
  4. 2025TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMscited by 50
  5. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  6. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  7. 2025Generative Frame Sampler for Long Video Understandingcited by 37
  8. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 37
  9. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  10. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 37
  11. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillationcited by 36
  12. 2025Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detectioncited by 34
  13. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  14. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  15. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  16. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  17. 2025VideoExplorer: Think With Videos For Agentic Long-Video Understandingcited by 29
  18. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 29
  19. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 27
  20. 2025LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoningcited by 27
  21. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  22. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 26
  23. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  24. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  25. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  26. 2025Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thoughtcited by 24
  27. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  28. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  29. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 22
  30. 2025VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluationcited by 21
  31. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 21
  32. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19
  33. 2025Multi-Agent System for Comprehensive Soccer Understandingcited by 18
  34. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  35. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 18
  36. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  37. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 16
  38. 2025REVISOR: Beyond Textual Reflection, Towards Multimodal Introspective Reasoning in Long-Form Video Understandingcited by 16

50 most cited of 220 All 220 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streams
  2. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging
  3. 2024Streaming Long Video Understanding with Large Language Models
  4. 2026Visual Token Coding for Video Multimodal Large Language Models
  5. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models
  6. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM
  7. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling
  8. 2025FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding