mlsanityData to 14 Oct 2026

Causal Diffusion Transformers for Generative Modeling

arXiv 2412.12095 ↗First posted 16 Dec 2024cs.CVcited by 31

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We introduce Causal Diffusion as the autoregressive (AR) counterpart of Diffusion models. It is a next-token(s) forecasting framework that is friendly to both discrete and continuous modalities and compatible with existing next-token prediction models like LLaMA and GPT. While recent works attempt to combine diffusion with AR models, we show that introducing sequential factorization to a diffusion model can substantially improve its performance and enables a smooth transition between AR and diffusion generation modes. Hence, we propose CausalFusion - a decoder-only transformer that dual-factorizes data across sequential tokens and diffusion noise levels, leading to state-of-the-art results on the ImageNet generation benchmark while also enjoying the AR advantage of generating an arbitrary number of tokens for in-context reasoning. We further demonstrate CausalFusion's multimodal capabilities through a joint image generation and captioning model, and showcase CausalFusion's ability for zero-shot in-context image manipulations. We hope that this work could provide the community with a fresh perspective on training multimodal models over discrete and continuous data.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 26 of its 75 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

26 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation10 Oct
  2. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction3 Apr
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2017Decoupled Weight Decay Regularizationcited by 1,831
  7. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  8. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  9. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  10. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  11. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 329
  12. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Modelscited by 254
10 more
  1. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  2. 2023Fast Training of Diffusion Models with Masked Transformerscited by 176
  3. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 155
  4. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  5. 2023Relay Diffusion: Unifying diffusion process across resolutions for image synthesiscited by 79
  6. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 66
  7. 2024MonoFormer: One Transformer for Both Diffusion and Autoregressioncited by 46
  8. 2024Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalizationcited by 36
  9. 2024FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Modelcited by 15
  10. 2024BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilitiescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 9.

31 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  2. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  3. 2025DDT: Decoupled Diffusion Transformercited by 109
  4. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 96
  5. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 94
  6. 2025PixNerd: Pixel Neural Field Diffusioncited by 82
  7. 2025Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generationcited by 71
  8. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 36
  9. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  10. 2025Next Block Prediction: Video Generation via Semi-Autoregressive Modelingcited by 26
  11. 2025A Survey of Interactive Generative Videocited by 23
  12. 2026INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modelingcited by 21
19 more
  1. 2025Latent Denoising Makes Good Tokenizerscited by 20
  2. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  3. 2025D-AR: Diffusion via Autoregressive Modelscited by 9
  4. 2025Video-GPT via Next Clip Diffusioncited by 8
  5. 2025Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generationcited by 7
  6. 2025Plug-and-Play Context Feature Reuse for Efficient Masked Generationcited by 5
  7. 2025Next Visual Granularity Generationcited by 5
  8. 2026Scalable Spatio-Temporal SE(3) Diffusion for Long-Horizon Protein Dynamicscited by 4
  9. 2026OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generationcited by 4
  10. 2025From Prediction to Perfection: Introducing Refinement to Autoregressive Image Generationcited by 3
  11. 2025MolSnap: Snap-Fast Molecular Generation with Latent Variational Mean Flowcited by 2
  12. 2026Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformerscited by 2
  13. 2026Rethinking Cross-Layer Information Routing in Diffusion Transformerscited by 2
  14. 2025DepMicroDiff: Diffusion-Based Dependency-Aware Multimodal Imputation for Microbiome Datacited by 1
  15. 2025CausalGeD: Blending Causality and Diffusion for Spatial Gene Expression Generationcited by 0
  16. 2026RepFlow: Representation Enhanced Flow Matching for Causal Effect Estimationcited by 0
  17. 2026Next Thoughts Are Distributions: Generative Autoregressive Reasoning in the Latent Spacecited by 0
  18. 2026World Motion Models: Flexible Sequence Modeling of SE(3) Trajectoriescited by 0
  19. 2026REACT: Rolling Denoising and Dual Decoupling for Reactive Robot Control with VLA Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models
  2. 2026Causal Autoregressive Diffusion Language Model
  3. 2026Causal Motion Diffusion Models for Autoregressive Motion Generation
  4. 2025MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation
  5. 2026Causality in Video Diffusers is Separable from Denoising
  6. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer
  7. 2025Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression
  8. 2026RepForcing: Representation-Space Initialization for Autoregressive Video Distillation