mlsanityData to 14 Oct 2026

Apollo: An Exploration of Video Understanding in Large Multimodal Models

arXiv 2412.10360 ↗First posted 13 Dec 2024cs.CV cs.AIcited by 89

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Despite the rapid integration of video perception capabilities into Large Multimodal Models (LMMs), the underlying mechanisms driving their video understanding remain poorly understood. Consequently, many design decisions in this domain are made without proper justification or analysis. The high computational cost of training and evaluating such models, coupled with limited open research, hinders the development of video-LMMs. To address this, we present a comprehensive study that helps uncover what effectively drives video understanding in LMMs. We begin by critically examining the primary contributors to the high computational requirements associated with video-LMM research and discover Scaling Consistency, wherein design and training decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. Leveraging these insights, we explored many video-specific aspects of video-LMMs, including video sampling, architectures, data composition, training schedules, and more. For example, we demonstrated that fps sampling during training is vastly preferable to uniform frame sampling and which vision encoders are the best for video representation. Guided by these findings, we introduce Apollo, a state-of-the-art family of LMMs that achieve superior performance across different model sizes. Our models can perceive hour-long videos efficiently, with Apollo-3B outperforming most existing 7B models with an impressive 55.1 on LongVideoBench. Apollo-7B is state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on Video-MME.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 63 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

52 of 63 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct
  2. 2024Exploring the Design Space of Visual Context Representation in Video MLLMs17 Oct
  3. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding22 Sep
  4. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  5. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos26 Aug
  6. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  7. 2024Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision8 Jul
  8. 2024OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understanding6 Jul
  9. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output3 Jul
  10. 2024Long Context Transfer from Language to Vision24 Jun
  11. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoning17 Jun
  12. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
5 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  2. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  3. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 35

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
23 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  5. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  6. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  7. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  8. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  9. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  10. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  11. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  12. 2024What matters when building vision-language models?cited by 326
  13. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  14. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  15. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  16. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  17. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  18. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  19. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  20. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  21. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  22. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  23. 2024FreeVA: Offline MLLM as Training-Free Video Assistantcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 51; 2026: 38.

89 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  2. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  3. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  4. 2025Scaling RL to Long Videoscited by 84
  5. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  6. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 65
  7. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  8. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 52
  9. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  10. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 46
  11. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  12. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
38 more
  1. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  2. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  3. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 30
  4. 2025AToken: A Unified Tokenizer for Visioncited by 30
  5. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 29
  6. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 29
  7. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  8. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 26
  9. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  10. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 23
  11. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Videocited by 21
  12. 2025Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?cited by 20
  13. 2025VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Managementcited by 19
  14. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 17
  15. 2025ARGUS: Hallucination and Omission Evaluation in Video-LLMscited by 16
  16. 2025Revisiting Multimodal Positional Encoding in Vision-Language Modelscited by 16
  17. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  18. 2025Chirality in Action: Time-Aware Video Representation Learning by Latent Straighteningcited by 11
  19. 2025Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understandingcited by 11
  20. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 11
  21. 2025Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenderscited by 10
  22. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  23. 2025Flow4Agent: Long-form Video Understanding via Motion Prior from Optical Flowcited by 9
  24. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 9
  25. 2026Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanismcited by 9
  26. 2025An LMM for Efficient Video Understanding via Reinforced Compression of Video Cubescited by 7
  27. 2025ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understandingcited by 7
  28. 2026SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoningcited by 7
  29. 2025From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignmentcited by 6
  30. 2025How Well Can General Vision-Language Models Learn Medicine By Watching Public Educational Videos?cited by 6
  31. 2025Failures to Surface Harmful Contents in Video Large Language Modelscited by 5
  32. 2026Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Modelscited by 5
  33. 2026Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisionscited by 5
  34. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 5
  35. 2025From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervisioncited by 4
  36. 2025ProfVLM: A lightweight video-language model for multi-view proficiency estimationcited by 4
  37. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  38. 2026Stateful Token Reduction for Long-Video Hybrid VLMscited by 4

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models
  2. 2025TinyLLaVA-Video: Towards Smaller LMMs for Video Understanding with Group Resampler
  3. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  4. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
  5. 2025Improving LLM Video Understanding with 16 Frames Per Second
  6. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  7. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding
  8. 2025VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation