# Mavors: Multi-granularity Video Representation for Multimodal Large Language Model

arXiv: https://arxiv.org/abs/2504.10068 · HTML: https://mlsanity.com/paper/2504.10068/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 110 of its 134 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 110)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)

## Built on by (most cited first, 30 of 32)

- [Monet: Reasoning in Latent Visual Space Beyond Images and Language](https://mlsanity.com/paper/2511.21395/) (2025)
- [Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration](https://mlsanity.com/paper/2505.20256/) (2025)
- [RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark](https://mlsanity.com/paper/2509.24897/) (2025)
- [VABench: A Comprehensive Benchmark for Audio-Video Generation](https://mlsanity.com/paper/2512.09299/) (2025)
- [AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration](https://mlsanity.com/paper/2510.10395/) (2025)
- [MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios](https://mlsanity.com/paper/2505.21333/) (2025)
- [BaseReward: A Strong Baseline for Multimodal Reward Model](https://mlsanity.com/paper/2509.16127/) (2025)
- [VidBridge-R1: Bridging QA and Captioning for RL-based Video Understanding Models with Intermediate Proxy Tasks](https://mlsanity.com/paper/2506.09079/) (2025)
- [VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining](https://mlsanity.com/paper/2603.15030/) (2026)
- [MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs](https://mlsanity.com/paper/2511.07250/) (2025)
- [BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents](https://mlsanity.com/paper/2602.12876/) (2026)
- [LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning](https://mlsanity.com/paper/2605.22012/) (2026)
- [Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?](https://mlsanity.com/paper/2604.03016/) (2026)
- [Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrieval](https://mlsanity.com/paper/2510.22765/) (2025)
- [Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos](https://mlsanity.com/paper/2605.18984/) (2026)
- [Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks](https://mlsanity.com/paper/2602.01630/) (2026)
- [OpenWorldLib: A Unified Codebase and Definition of Advanced World Models](https://mlsanity.com/paper/2604.04707/) (2026)
- [Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization](https://mlsanity.com/paper/2605.10780/) (2026)
- [Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Model](https://mlsanity.com/paper/2605.12163/) (2026)
- [ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships](https://mlsanity.com/paper/2607.14681/) (2026)
- [OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models](https://mlsanity.com/paper/2608.03812/) (2026)
- [The Escalator Problem: Identifying Implicit Motion Blindness in AI for Accessibility](https://mlsanity.com/paper/2508.07989/) (2025)
- [MorphoBench: A Benchmark with Difficulty Adaptive to Model Reasoning](https://mlsanity.com/paper/2510.14265/) (2025)
- [KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation](https://mlsanity.com/paper/2607.14202/) (2026)
- [Beacon: Knowing When and How to Perform Agentic Visual Reasoning](https://mlsanity.com/paper/2607.28595/) (2026)
- [Hybrid Attribution Priors for Explainable and Robust Model Training](https://mlsanity.com/paper/2512.14719/) (2025)
- [RefCaptioner: Multi-Reference Image-Grounded Video Captioning](https://mlsanity.com/paper/2607.28509/) (2026)
- [AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward](https://mlsanity.com/paper/2608.06930/) (2026)
- [VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding](https://mlsanity.com/paper/2609.15606/) (2026)
- [Think Before You Score: Thinking Reward Model for Visual Generation](https://mlsanity.com/paper/2609.37372/) (2026)

## Abstract (arXiv)

Long-context video understanding in multimodal large language models (MLLMs) faces a critical challenge: balancing computational efficiency with the retention of fine-grained spatio-temporal patterns. Existing approaches (e.g., sparse sampling, dense sampling with low resolution, and token compression) suffer from significant information loss in temporal dynamics, spatial details, or subtle interactions, particularly in videos with complex motion or varying resolutions. To address this, we propose Mavors, a novel framework that introduces Multi-granularity video representation for holistic long-video modeling. Specifically, Mavors directly encodes raw video content into latent representations through two core components: 1) an Intra-chunk Vision Encoder (IVE) that preserves high-resolution spatial features via 3D convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator (IFA) that establishes temporal coherence across chunks using transformer-based dependency modeling with chunk-level rotary position encodings. Moreover, the framework unifies image and video understanding by treating images as single-frame videos via sub-image decomposition. Experiments across diverse benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity and temporal continuity, significantly outperforming existing methods in tasks requiring fine-grained spatio-temporal reasoning.