# Unleashing Hour-Scale Video Training for Long Video-Language Understanding

arXiv: https://arxiv.org/abs/2506.05332 · HTML: https://mlsanity.com/paper/2506.05332/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 60 of its 70 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 60)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)

## Built on by (most cited first, 23 of 23)

- [VideoExplorer: Think With Videos For Agentic Long-Video Understanding](https://mlsanity.com/paper/2506.10821/) (2025)
- [LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoning](https://mlsanity.com/paper/2509.24786/) (2025)
- [ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesis](https://mlsanity.com/paper/2509.23652/) (2025)
- [Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously](https://mlsanity.com/paper/2603.12262/) (2026)
- [TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learning](https://mlsanity.com/paper/2511.05489/) (2025)
- [VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Management](https://mlsanity.com/paper/2512.04540/) (2025)
- [VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking](https://mlsanity.com/paper/2603.20185/) (2026)
- [REVISOR: Beyond Textual Reflection, Towards Multimodal Introspective Reasoning in Long-Form Video Understanding](https://mlsanity.com/paper/2511.13026/) (2025)
- [EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use](https://mlsanity.com/paper/2602.15329/) (2026)
- [Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data](https://mlsanity.com/paper/2509.03501/) (2025)
- [Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoning](https://mlsanity.com/paper/2510.15440/) (2025)
- [Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding](https://mlsanity.com/paper/2608.25356/) (2026)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [Dynamic Hub-and-Spoke Memory for Streaming Video Understanding](https://mlsanity.com/paper/2608.30294/) (2026)
- [Routing Before Looking: Query-Adaptive Evidence Acquisition for Long-form Video Understanding](https://mlsanity.com/paper/2608.20805/) (2026)
- [Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2510.24870/) (2025)
- [Evaluating Multimodal Narrative Understanding of Popular Hollywood Films](https://mlsanity.com/paper/2608.21430/) (2026)
- [K9-Bench: Evaluating Multimodal LLMs on Canine-Centric Videos](https://mlsanity.com/paper/2607.02680/) (2026)
- [VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding](https://mlsanity.com/paper/2609.15606/) (2026)
- [VideoResearcher: Self-Improving Tool Design for Long-Video Understanding](https://mlsanity.com/paper/2609.19664/) (2026)
- [VideoLoop: Looped Working Memory Against Semantic Thrashing in Long-Form Video Agents](https://mlsanity.com/paper/2609.38119/) (2026)
- [VideoEvoHarness: Self Evolving Agent Harness from Unlabeled Videos for Video Understanding](https://mlsanity.com/paper/2610.15389/) (2026)
- [Towards Cost-Efficient Video Dataset Construction with Large Language Models](https://mlsanity.com/paper/2610.16857/) (2026)

## Abstract (arXiv)

Recent long-form video-language understanding benchmarks have driven progress in video large multimodal models (Video-LMMs). However, the scarcity of well-annotated long videos has left the training of hour-long Video-LMMs underexplored. To close this gap, we present VideoMarathon, a large-scale hour-long video instruction-following dataset. This dataset includes around 9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60 minutes per video. Specifically, it contains 3.3M high-quality QA pairs, spanning six fundamental topics: temporality, spatiality, object, action, scene, and event. Compared to existing video instruction datasets, VideoMarathon significantly extends training video durations up to 1 hour, and supports 22 diverse tasks requiring both short- and long-term video comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and efficient Video-LMM for hour-scale video-language modeling. It enables hour-long video training and inference at 1-FPS sampling by leveraging a memory augmentation module, which adaptively integrates question-relevant and spatiotemporally informative semantics from the cached full video context. In our experiments, Hour-LLaVA achieves the best performance on multiple representative long video-language benchmarks, demonstrating the high quality of the VideoMarathon dataset and the superiority of the Hour-LLaVA model.