mlsanityData to 7 Oct 2026

Video understanding with MLLMs

742 papers2 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: video, understanding, temporal, video understanding, videos

Key facts. Video understanding with MLLMs: 742 papers on arXiv since 2022 (data through 7 Oct 2026). 147 of its 742 mapped papers (20%) cite “Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis” (2024, arXiv 2405.21075). 112 of its 742 mapped papers (15%) cite “MVBench: A Comprehensive Multi-modal Video Understanding Benchmark” (2023, arXiv 2311.17005). 27 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

27 new papers, 17 Sep – 7 Oct; 2 this week.

  1. This week · 2
  2. VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction
  3. ReMem: Streaming Video Understanding With Long Context Retention
  4. Earlier · 25
  5. Representation--Behavior Alignment for Explainable Weakly-Supervised Video Anomaly Detection
  6. Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency
  7. Behavior Pack Optimization for Video MLLM Post-Training
  8. FlashBack: Knowing When to Remember in Streaming Vision-Language Models
  9. Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics
  10. FORTE: Adaptive Scoring and Exact Keyframe Selection for Long-Video Question Answering
  11. Less Data, Better Timing: Student-Curriculum Coupling for VLM On-Policy Distillation in Temporal Video Grounding
  12. Frame Differential On-Policy Self-Distillation for Video Reasoning
All 27
  1. PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation also fits: Visual token pruning and efficiency
  2. ThinkV2V: Unleashing the Reasoning Capability of MLLMs for Instruction-Guided Video Editing
  3. VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding
  4. LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension
  5. Beyond Binary Preferences: Graded Preference Optimization for Limb-Motion Captioning also fits: Prompt learning and adaptation of CLIP
  6. MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms
  7. Counterfactual Attention Policy Distillation for Temporal Video Grounding
  8. PARSEE-VAD: Efficient Training-Free Online Video Anomaly Detection via Proposition-Aware Reasoning and Streaming Evidence Escalation
  9. ReVA: A Scene-Centric Dataset Beyond Repetition for Remote Sensing Video Question Answering also fits: Embodied and 3D vision-language models
  10. Sprout: Building Dynamic Memory While Reasoning for Agentic Video Understanding
  11. ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models also fits: Embodied and 3D vision-language models
  12. MetaSampling: Making Frame Samplers Efficient for Long-Video Question Answering
  13. KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding
  14. MVVBench: Benchmarking 4D Reasoning in Vision-Language Models also fits: VLM benchmarks and visual reasoning
  15. MoVISA: Multi-Token Reasoning for Video Object Segmentation
  16. CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models
  17. Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows also fits: VLM benchmarks and visual reasoning

Foundations

Most cited by papers here

  1. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis2024cited by 147 here
  2. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark2023cited by 112 here
  3. Video-LLaVA: Learning United Visual Representation by Alignment Before Projection2023cited by 97 here
  4. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding2023cited by 92 here
  5. LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding2024cited by 81 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 60 here
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 92 here
  3. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 50 here
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 147 here
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 78 here
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 46 here
18 more it builds on, not in this chain
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 41 here
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 52 here
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 97 here
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 112 here
  5. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 53 here
  6. 2023VILA: On Pre-training for Visual Language Modelscited by 33 here
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 56 here
  8. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 43 here
  9. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 57 here
  10. 2024Long Context Transfer from Language to Visioncited by 54 here
  11. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 81 here
  12. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 33 here
  13. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 95 here
  14. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 35 here
  15. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 66 here
  16. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 37 here
  17. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 53 here
  18. 2025Qwen3-VL Technical Reportcited by 50 here

The 24 most cited of 1,024.

Browse

30 of 742, ranked 6–35

  1. 6LVBench: An Extreme Long Video Understanding Benchmark202457
  2. 7Long Context Transfer from Language to Vision202454
  3. 8Video-R1: Reinforcing Video Reasoning in MLLMs202553
  4. 9LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models202353
  5. 10TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding202350
  6. 11PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning202443
  7. 12LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding202437
  8. 13LongVILA: Scaling Long-Context Visual Language Models for Long Videos202435
  9. 14Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding202427
  10. 15VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling202424
All 30
  1. 16InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling202522
  2. 17Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input202422
  3. 18mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models202421
  4. 19MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding202420
  5. 20TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning202420
  6. 21Adaptive Keyframe Sampling for Long Video Understanding202519
  7. 22Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding202519
  8. 23LITA: Language Instructed Temporal-Localization Assistant202419
  9. 24Scaling RL to Long Videos202518
  10. 25Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning202518
  11. 26VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection202418
  12. 27VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding202417
  13. 28VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding202417
  14. 29MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding202416
  15. 30MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens202415
  16. 31Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning202315
  17. 32BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding202513
  18. 33CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding202413
  19. 34WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs202512
  20. 35M-LLM Based Video Frame Selection for Efficient Video Understanding202511