mlsanityData to 7 Oct 2026

Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers

arXiv 2402.19479 ↗First posted 29 Feb 2024cs.CVcited by 389

Not on the map. Why?

Cited by 100 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The quality of the data and annotation upper-bounds the quality of a downstream model. While there exist large text corpora and image-text pairs, high-quality video-text data is much harder to collect. First of all, manual labeling is more time-consuming, as it requires an annotator to watch an entire video. Second, videos have a temporal dimension, consisting of several scenes stacked together, and showing multiple actions. Accordingly, to establish a video dataset with high-quality captions, we propose an automatic approach leveraging multimodal inputs, such as textual video description, subtitles, and individual video frames. Specifically, we curate 3.8M high-resolution videos from the publicly available HD-VILA-100M dataset. We then split them into semantically consistent video clips, and apply multiple cross-modality teacher models to obtain captions for each video. Next, we finetune a retrieval model on a small subset where the best caption of each video is manually selected and then employ the model in the whole dataset to select the best caption as the annotation. In this way, we get 70M videos paired with high-quality text captions. We dub the dataset as Panda-70M. We show the value of the proposed dataset on three downstream tasks: video captioning, video and text retrieval, and text-driven video generation. The models trained on the proposed data score substantially better on the majority of metrics across all the tasks.

Key facts. Cited by 389 papers in the map (data through 7 Oct 2026). Builds on 82 of its 94 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

82 of 94 references.

References · 82

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  11. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  13. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  14. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  15. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  16. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  17. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  18. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  19. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  20. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  21. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  22. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  23. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  24. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  25. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  26. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  27. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  28. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  29. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  30. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  31. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  32. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  33. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  34. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  35. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  36. 2017Localizing Moments in Video with Natural Languagecited by 660
  37. 2023ModelScope Text-to-Video Technical Reportcited by 651
  38. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 116; 2025: 186; 2026: 87.

389 papers.

Sort:

By area:

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  2. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  6. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  7. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  8. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  9. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  10. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  11. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  12. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
38 more
  1. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  2. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  3. 2025ViPE: Video Pose Engine for 3D Geometric Perceptioncited by 162
  4. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  5. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  6. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  7. 2026Advancing Open-source World Modelscited by 144
  8. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  9. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  10. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  11. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  12. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  13. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  14. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  15. 2025Phantom: Subject-consistent video generation via cross-modal alignmentcited by 126
  16. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  17. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  18. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  19. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  20. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  21. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  22. 2025Controllable Video Generation: A Surveycited by 93
  23. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  24. 2024FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attentioncited by 87
  25. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  26. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  27. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  28. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
  29. 2024EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagationcited by 76
  30. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  31. 2025Improving the Diffusability of Autoencoderscited by 74
  32. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  33. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  34. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  35. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  36. 2024Vript: A Video Is Worth Thousands of Wordscited by 71
  37. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  38. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69

50 most cited of 389 All 389 (JSON)