# LVD-2M: A Long-take Video Dataset with Temporally Dense Captions

arXiv: https://arxiv.org/abs/2410.10816 · HTML: https://mlsanity.com/paper/2410.10816/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text](https://mlsanity.com/paper/2403.14773/) (2024)
- [Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions](https://mlsanity.com/paper/2111.10337/) (2021)
- [Revisiting the "Video" in Video-Language Understanding](https://mlsanity.com/paper/2206.01720/) (2022)
- [Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation](https://mlsanity.com/paper/2305.10874/) (2023)
- [Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising](https://mlsanity.com/paper/2305.18264/) (2023)

## Built on by (most cited first, 25 of 25)

- [UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions](https://mlsanity.com/paper/2506.13691/) (2025)
- [Mind the Time: Temporally-Controlled Multi-Event Video Generation](https://mlsanity.com/paper/2412.05263/) (2024)
- [Mavors: Multi-granularity Video Representation for Multimodal Large Language Model](https://mlsanity.com/paper/2504.10068/) (2025)
- [CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Models](https://mlsanity.com/paper/2508.11484/) (2025)
- [VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation](https://mlsanity.com/paper/2502.12782/) (2025)
- [VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation](https://mlsanity.com/paper/2503.01739/) (2025)
- [ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videos](https://mlsanity.com/paper/2503.16400/) (2025)
- [TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos](https://mlsanity.com/paper/2505.20124/) (2025)
- [MuSS: A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation](https://mlsanity.com/paper/2604.23789/) (2026)
- [AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation](https://mlsanity.com/paper/2506.03126/) (2025)
- [FrameOracle: Learning What to See and How Much to See in Videos](https://mlsanity.com/paper/2510.03584/) (2025)
- [SViM3D: Stable Video Material Diffusion for Single Image 3D Generation](https://mlsanity.com/paper/2510.08271/) (2025)
- [CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation](https://mlsanity.com/paper/2606.09639/) (2026)
- [WorldOlympiad: Can Your World Model Survive a Triathlon?](https://mlsanity.com/paper/2606.11129/) (2026)
- [FakeParts: a New Family of AI-Generated DeepFakes](https://mlsanity.com/paper/2508.21052/) (2025)
- [HAIC: Improving Human Action Understanding and Generation with Better Captions for Multi-modal Large Language Models](https://mlsanity.com/paper/2502.20811/) (2025)
- [VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.21556/) (2025)
- [FinCap: Topic-Aligned Captions for Short-Form Financial YouTube Videos](https://mlsanity.com/paper/2509.25745/) (2025)
- [CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning](https://mlsanity.com/paper/2605.26967/) (2026)
- [MoVA: Learning Asymmetric Dual Projections for Modular Long Video-Text Alignment](https://mlsanity.com/paper/2607.00858/) (2026)
- [LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training](https://mlsanity.com/paper/2608.24845/) (2026)
- [TempCloze: Can Video-LLMs Identify the Missing Middle?](https://mlsanity.com/paper/2609.01515/) (2026)
- [From Static to Dynamic: On-Policy Distillation from Image to Video Diffusion Models](https://mlsanity.com/paper/2609.34371/) (2026)
- [LongTake: Learning to Sustain Dynamics in Long-Horizon Video Generation](https://mlsanity.com/paper/2609.38562/) (2026)
- [Token-Level Video Reinforcement Learning](https://mlsanity.com/paper/2610.01973/) (2026)

## Abstract (arXiv)

The efficacy of video generation models heavily depends on the quality of their training datasets. Most previous video generation models are trained on short video clips, while recently there has been increasing interest in training long video generation models directly on longer videos. However, the lack of such high-quality long videos impedes the advancement of long video generation. To promote research in long video generation, we desire a new dataset with four key features essential for training long video generation models: (1) long videos covering at least 10 seconds, (2) long-take videos without cuts, (3) large motion and diverse contents, and (4) temporally dense captions. To achieve this, we introduce a new pipeline for selecting high-quality long-take videos and generating temporally dense captions. Specifically, we define a set of metrics to quantitatively assess video quality including scene cuts, dynamic degrees, and semantic-level quality, enabling us to filter high-quality long-take videos from a large amount of source videos. Subsequently, we develop a hierarchical video captioning pipeline to annotate long videos with temporally-dense captions. With this pipeline, we curate the first long-take video dataset, LVD-2M, comprising 2 million long-take videos, each covering more than 10 seconds and annotated with temporally dense captions. We further validate the effectiveness of LVD-2M by fine-tuning video generation models to generate long videos with dynamic motions. We believe our work will significantly contribute to future research in long video generation.