# Progressive Video Condensation with MLLM Agent for Long-form Video Understanding

arXiv: https://arxiv.org/abs/2604.02891 · HTML: https://mlsanity.com/paper/2604.02891/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [Self-Chained Image-Language Model for Video Localization and Question Answering](https://mlsanity.com/paper/2305.06988/) (2023)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [Tarsier: Recipes for Training and Evaluating Large Video Description Models](https://mlsanity.com/paper/2407.00634/) (2024)
- [An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM](https://mlsanity.com/paper/2403.18406/) (2024)
- [Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA](https://mlsanity.com/paper/2406.09396/) (2024)
- [DrVideo: Document Retrieval Based Long Video Understanding](https://mlsanity.com/paper/2406.12846/) (2024)
- [Language Repository for Long Video Understanding](https://mlsanity.com/paper/2403.14622/) (2024)
- [TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering](https://mlsanity.com/paper/2404.01476/) (2024)
- [VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understanding](https://mlsanity.com/paper/2512.12360/) (2025)
- [Imp: Highly Capable Large Multimodal Models for Mobile Devices](https://mlsanity.com/paper/2405.12107/) (2024)
- [Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Models](https://mlsanity.com/paper/2503.14075/) (2025)

## Built on by (most cited first, 2 of 2)

- [CADER: Confidence-Aware Dynamic Evidence Reasoning for Long-Video Understanding](https://mlsanity.com/paper/2607.24582/) (2026)
- [InSight-doc: Agentic Visual Perception for Long-Document Understanding](https://mlsanity.com/paper/2608.10628/) (2026)

## Abstract (arXiv)

Understanding long videos requires extracting query-relevant information from long sequences under tight compute budgets. Existing text-then-LLM pipelines lose fine-grained visual cues, while video-based multimodal large language models (MLLMs) can keep visual details but are too frame-hungry and computationally expensive. In this work, we aim to harness MLLMs for efficient video understanding. We propose ProVCA, a progressive video condensation agent that iteratively locates key video frames at multiple granularities. ProVCA first adopts a segment localization module to identify the video segment relevant to the query, then a snippet selection module to select important snippets based on similarity, and finally a keyframe refinement module to pinpoint specific keyframes in those snippets. By progressively narrowing the scope from coarse segments to fine frames, ProVCA identifies a small set of keyframes for MLLM-based reasoning. ProVCA achieves state-of-the-art zero-shot accuracies of 69.3% on EgoSchema, 80.5% on NExT-QA, and 77.7% on IntentQA, while using fewer frames than previous training-free methods.