mlsanityData to 7 Oct 2026

Training Video Foundation Models with NVIDIA NeMo

arXiv 2503.12964 ↗First posted 17 Mar 2025cs.CV cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Video Foundation Models (VFMs) have recently been used to simulate the real world to train physical AI systems and develop creative visual experiences. However, there are significant challenges in training large-scale, high quality VFMs that can generate high-quality videos. We present a scalable, open-source VFM training pipeline with NVIDIA NeMo, providing accelerated video dataset curation, multimodal data loading, and parallelized video diffusion model training and inference. We also provide a comprehensive performance analysis highlighting best practices for efficient VFM training and inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

22 of 27 references.

References · 22

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  8. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  10. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  12. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
10 more
  1. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  2. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  3. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  4. 2024xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelismcited by 55
  5. 2024Scaling Laws For Diffusion Transformerscited by 41
  6. 2024On the Scalability of Diffusion-based Text-to-Image Generationcited by 36
  7. 2024Simple and Fast Distillation of Diffusion Modelscited by 31
  8. 2023Improved Order Analysis and Design of Exponential Integrator for Diffusion Models Samplingcited by 16
  9. 2024DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformerscited by 15
  10. 2024Scaling Properties of Diffusion Models for Perceptual Taskscited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models
  2. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities
  3. 2026Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models
  4. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation
  5. 2025VFMF: World Modeling by Forecasting Vision Foundation Model Features
  6. 2025BIFE: Better Interaction, Fewer Errors for Minute-Long Video Generation
  7. 2026Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models
  8. 2025From Pretrain to Pain: Adversarial Vulnerability of Video Foundation Models Without Task Knowledge