mlsanityData to 7 Oct 2026

MarDini: Masked Autoregressive Diffusion for Video Generation at Scale

arXiv 2410.20280 ↗First posted 26 Oct 2024cs.CV cs.AIcited by 49

On the map: Diffusion models → Video generation

Abstract (arXiv)

We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses on spatial generation in an asymmetric network design: i) a MAR-based planning model containing most of the parameters generates planning signals for each masked frame using low-resolution input; ii) a lightweight generation model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini's MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the computational resources to the low-resolution planning model, making computationally expensive but important spatio-temporal attention feasible at scale. MarDini sets a new state-of-the-art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 94 of its 116 references that are in the map. First posted on arXiv 26 Oct 2024.

Builds on

94 of 116 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation27 Aug
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  3. 2024Video Interpolation with Diffusion Models1 Apr
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  5. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models28 Nov
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  7. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  8. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  9. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
  10. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing9 Oct
  11. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation9 Oct
  12. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
9 more
  1. 2023ModelScope Text-to-Video Technical Report12 Aug
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  5. 2023LDMVFI: Video Frame Interpolation with Latent Diffusion Models16 Mar
  6. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  9. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May

Other references · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2024The Llama 3 Herd of Modelscited by 14,192
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  11. 2020Denoising Diffusion Implicit Modelscited by 9,655
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2016Layer Normalizationcited by 5,579
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  13. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  14. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  15. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  16. 2019Root Mean Square Layer Normalizationcited by 1,425
  17. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  18. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  19. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  20. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  21. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  22. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  23. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  24. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  25. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  26. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  27. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  28. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  29. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  30. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  31. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  32. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  33. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  34. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  35. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  36. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  37. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  38. 2022MAGVIT: Masked Generative Video Transformercited by 341

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 29; 2026: 15.

49 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  3. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  4. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 209
  5. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  6. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 91
  7. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 72
  8. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 45
  9. 2024Autoregressive Models in Vision: A Surveycited by 39
  10. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  11. 2025VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual interventioncited by 33
  12. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 23
37 more
  1. 2025Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspectivecited by 22
  2. 2024Towards Precise Scaling Laws for Video Diffusion Transformerscited by 20
  3. 2026INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modelingcited by 18
  4. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  5. 2026Streaming Autoregressive Video Generation via Diagonal Distillationcited by 15
  6. 2025Ingredients: Blending Custom Photos with Video Diffusion Transformerscited by 13
  7. 2025BAgger: Backwards Aggregation for Mitigating Drift in Autoregressive Video Diffusion Modelscited by 13
  8. 2025Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Thinkcited by 12
  9. 2026MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Enginescited by 12
  10. 2024Video Diffusion Transformers are In-Context Learnerscited by 11
  11. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 11
  12. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  13. 2025Continuous Visual Autoregressive Generation via Score Maximizationcited by 9
  14. 2025Scaling Zero-Shot Reference-to-Video Generationcited by 9
  15. 2025Scaling Sequence-to-Sequence Generative Neural Renderingcited by 8
  16. 2025ConfRover: Simultaneous Modeling of Protein Conformation and Dynamics via Autoregressioncited by 7
  17. 2025AirScape: An Aerial Generative World Model with Motion Controllabilitycited by 7
  18. 2025Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusioncited by 7
  19. 2025WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusioncited by 6
  20. 2025UniVid: Unifying Vision Tasks with Pre-trained Video Generation Modelscited by 5
  21. 2026Causality in Video Diffusers is Separable from Denoisingcited by 4
  22. 2026EditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editingcited by 4
  23. 2025NeoBabel: A Multilingual Open Tower for Visual Generationcited by 2
  24. 2025Mixture of States: Routing Token-Level Dynamics for Multimodal Generationcited by 2
  25. 2026SoulX-LiveAct: Towards Hour-Scale Real-Time Human Animation with Neighbor Forcing and ConvKV Memorycited by 2
  26. 2024VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual interventioncited by 1
  27. 2025MARRS: Masked Autoregressive Unit-based Reaction Synthesiscited by 1
  28. 2026Image-to-Video Diffusion: From Foundations to Open Frontierscited by 1
  29. 2026Stream Forcing: Constructing Unified Training Trajectory for Robust Streaming Video Generationcited by 1
  30. 2025Capturing Conditional Dependence via Auto-regressive Diffusion Modelscited by 0
  31. 2025Controllable Coupled Image Generation via Diffusion Modelscited by 0
  32. 2026Lumos-Nexus: Efficient Frequency Bridging with Homogeneous Latent Space for Video Unified Modelscited by 0
  33. 2026C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Spacecited by 0
  34. 2026Recency Forcing: Bridging the Long-Horizon Gap in Autoregressive Video Generationcited by 0
  35. 2026ViRDM: Taming Representation Distribution Matching for Few-Step Causal Video Generationcited by 0
  36. 2026SoL-Refiner: Speed-of-Light One-Step Refinement for High-Resolution Videocited by 0
  37. 2026Unveiling the Value of Motion for Cinematic Camera Trajectoriescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion
  2. 2026Efficient Video Diffusion Models: Advancements and Challenges
  3. 2022Video Diffusion Models
  4. 2024Progressive Autoregressive Video Diffusion Models
  5. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
  6. 2022Diffusion Models for Video Prediction and Infilling
  7. 2026Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout
  8. 2025Playing with Transformer at 30+ FPS via Next-Frame Diffusion