mlsanityData to 14 Oct 2026

InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling

arXiv 2501.12386 ↗First posted 21 Jan 2025cs.CVcited by 211

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

This paper aims to improve the performance of video multimodal large language models (MLLM) via long and rich context (LRC) modeling. As a result, we develop a new version of InternVideo2.5 with a focus on enhancing the original MLLMs' ability to perceive fine-grained details and capture long-form temporal structure in videos. Specifically, our approach incorporates dense vision task annotations into MLLMs using direct preference optimization and develops compact spatiotemporal representations through adaptive hierarchical token compression. Experimental results demonstrate this unique design of LRC greatly improves the results of video MLLM in mainstream video understanding benchmarks (short & long), enabling the MLLM to memorize significantly longer video inputs (at least 6x longer than the original), and master specialized vision capabilities like object tracking and segmentation. Our work highlights the importance of multimodal context richness (length and fineness) in empowering MLLM's innate abilites (focus and memory), providing new insights for future research on video MLLM. Code and models are available at https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2.5

Key facts. Cited by 211 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 92 of its 105 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

92 of 105 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling31 Dec
  2. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Models13 Dec
  3. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct
  4. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding22 Sep
  5. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture4 Sep
  6. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos26 Aug
  7. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  8. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models9 Aug
  9. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  10. 2024Long Context Transfer from Language to Vision24 Jun
  11. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding20 Jun
  12. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoning17 Jun
7 more
  1. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding13 Jun
  2. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  4. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding18 Mar
  5. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  6. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov

Other references · 73

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2017The Kinetics Human Action Video Datasetcited by 1,909
  11. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
38 more
  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  3. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  4. 2022Planning-oriented Autonomous Drivingcited by 1,087
  5. 2016Modeling Context in Referring Expressionscited by 1,063
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  7. 2022Token Merging: Your ViT But Fastercited by 1,021
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  9. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  11. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  12. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  13. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  14. 2017Localizing Moments in Video with Natural Languagecited by 663
  15. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  16. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  17. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 568
  18. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  19. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  20. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  21. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  22. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  23. 2018SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networkscited by 446
  24. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  25. 2024InternLM2 Technical Reportcited by 397
  26. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  27. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  28. 2018ATOM: Accurate Tracking by Overlap Maximizationcited by 285
  29. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  30. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
  31. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  32. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  33. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  34. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 190
  35. 2022Language as Queries for Referring Video Object Segmentationcited by 182
  36. 2019SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelinescited by 177
  37. 2023FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Leadcited by 175
  38. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 109; 2026: 102.

211 papers.

Sort:

By area:

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  2. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
  3. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  4. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98
  5. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  6. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  7. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65
  8. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  9. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 49
  10. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 47
  11. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 46
  12. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 42
38 more
  1. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  2. 2025Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Modelscited by 38
  3. 2025Generative Frame Sampler for Long Video Understandingcited by 37
  4. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  5. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 37
  6. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 36
  7. 2025BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanationcited by 33
  8. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  9. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  10. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 30
  11. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 28
  12. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 28
  13. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 28
  14. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  15. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 26
  16. 2025Streaming Video Instruction Tuningcited by 26
  17. 2025VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videoscited by 25
  18. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  19. 2025LongVideoAgent: Multi-Agent Reasoning with Long Videoscited by 25
  20. 2025Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thoughtcited by 24
  21. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  22. 2026Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoningcited by 24
  23. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  24. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 23
  25. 2025Nexar Dashcam Collision Prediction Dataset and Challengecited by 21
  26. 2025VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluationcited by 21
  27. 2025Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?cited by 20
  28. 2025Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?cited by 20
  29. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 18
  30. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 16
  31. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
  32. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
  33. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  34. 2025Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thoughtcited by 14
  35. 2025ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Modelscited by 13
  36. 2025WikiVideo: Article Generation from Multiple Videoscited by 13
  37. 2025SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Modelscited by 13
  38. 2025RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligencecited by 13

50 most cited of 211 All 211 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Design Space of Visual Context Representation in Video MLLMs
  2. 2026Learning Compact Video Representations for Efficient Long-form Video Understanding in Large Multimodal Models
  3. 2025Iterative Zoom-In: Temporal Interval Exploration for Long Video Understanding
  4. 2025Test-Time Temporal Sampling for Efficient MLLM Video Understanding
  5. 2024Visual Context Window Extension: A New Perspective for Long Video Understanding
  6. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  7. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
  8. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding