# Causal Diffusion Transformers for Generative Modeling

arXiv: https://arxiv.org/abs/2412.12095 · HTML: https://mlsanity.com/paper/2412.12095/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 26 of its 75 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://mlsanity.com/paper/2408.11039/) (2024)
- [PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation](https://mlsanity.com/paper/2403.04692/) (2024)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)
- [Fast Training of Diffusion Models with Masked Transformers](https://mlsanity.com/paper/2306.09305/) (2023)
- [Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation](https://mlsanity.com/paper/2409.04410/) (2024)
- [Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers](https://mlsanity.com/paper/2405.05945/) (2024)
- [Relay Diffusion: Unifying diffusion process across resolutions for image synthesis](https://mlsanity.com/paper/2309.03350/) (2023)
- [Scaling Diffusion Transformers to 16 Billion Parameters](https://mlsanity.com/paper/2407.11633/) (2024)
- [MonoFormer: One Transformer for Both Diffusion and Autoregression](https://mlsanity.com/paper/2409.16280/) (2024)
- [Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalization](https://mlsanity.com/paper/2406.09416/) (2024)
- [DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation](https://mlsanity.com/paper/2410.08159/) (2024)
- [FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model](https://mlsanity.com/paper/2410.13925/) (2024)
- [BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities](https://mlsanity.com/paper/2410.14672/) (2024)

## Built on by (most cited first, 30 of 31)

- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [DDT: Decoupled Diffusion Transformer](https://mlsanity.com/paper/2504.05741/) (2025)
- [Masked Autoencoders Are Effective Tokenizers for Diffusion Models](https://mlsanity.com/paper/2502.03444/) (2025)
- [SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer](https://mlsanity.com/paper/2509.24695/) (2025)
- [PixNerd: Pixel Neural Field Diffusion](https://mlsanity.com/paper/2507.23268/) (2025)
- [Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2502.20388/) (2025)
- [AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models](https://mlsanity.com/paper/2509.25162/) (2025)
- [Energy-Based Transformers are Scalable Learners and Thinkers](https://mlsanity.com/paper/2507.02092/) (2025)
- [Next Block Prediction: Video Generation via Semi-Autoregressive Modeling](https://mlsanity.com/paper/2502.07737/) (2025)
- [A Survey of Interactive Generative Video](https://mlsanity.com/paper/2504.21853/) (2025)
- [INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modeling](https://mlsanity.com/paper/2604.07209/) (2026)
- [Latent Denoising Makes Good Tokenizers](https://mlsanity.com/paper/2507.15856/) (2025)
- [Position: Interactive Generative Video as Next-Generation Game Engine](https://mlsanity.com/paper/2503.17359/) (2025)
- [D-AR: Diffusion via Autoregressive Models](https://mlsanity.com/paper/2505.23660/) (2025)
- [Video-GPT via Next Clip Diffusion](https://mlsanity.com/paper/2505.12489/) (2025)
- [Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation](https://mlsanity.com/paper/2510.08994/) (2025)
- [Plug-and-Play Context Feature Reuse for Efficient Masked Generation](https://mlsanity.com/paper/2505.19089/) (2025)
- [Next Visual Granularity Generation](https://mlsanity.com/paper/2508.12811/) (2025)
- [Scalable Spatio-Temporal SE(3) Diffusion for Long-Horizon Protein Dynamics](https://mlsanity.com/paper/2602.02128/) (2026)
- [OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation](https://mlsanity.com/paper/2605.30519/) (2026)
- [From Prediction to Perfection: Introducing Refinement to Autoregressive Image Generation](https://mlsanity.com/paper/2505.16324/) (2025)
- [MolSnap: Snap-Fast Molecular Generation with Latent Variational Mean Flow](https://mlsanity.com/paper/2508.05411/) (2025)
- [Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformers](https://mlsanity.com/paper/2603.27666/) (2026)
- [Rethinking Cross-Layer Information Routing in Diffusion Transformers](https://mlsanity.com/paper/2605.20708/) (2026)
- [DepMicroDiff: Diffusion-Based Dependency-Aware Multimodal Imputation for Microbiome Data](https://mlsanity.com/paper/2507.23676/) (2025)
- [CausalGeD: Blending Causality and Diffusion for Spatial Gene Expression Generation](https://mlsanity.com/paper/2502.07751/) (2025)
- [RepFlow: Representation Enhanced Flow Matching for Causal Effect Estimation](https://mlsanity.com/paper/2605.05890/) (2026)
- [Next Thoughts Are Distributions: Generative Autoregressive Reasoning in the Latent Space](https://mlsanity.com/paper/2609.33271/) (2026)
- [World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories](https://mlsanity.com/paper/2610.01742/) (2026)

## Abstract (arXiv)

We introduce Causal Diffusion as the autoregressive (AR) counterpart of Diffusion models. It is a next-token(s) forecasting framework that is friendly to both discrete and continuous modalities and compatible with existing next-token prediction models like LLaMA and GPT. While recent works attempt to combine diffusion with AR models, we show that introducing sequential factorization to a diffusion model can substantially improve its performance and enables a smooth transition between AR and diffusion generation modes. Hence, we propose CausalFusion - a decoder-only transformer that dual-factorizes data across sequential tokens and diffusion noise levels, leading to state-of-the-art results on the ImageNet generation benchmark while also enjoying the AR advantage of generating an arbitrary number of tokens for in-context reasoning. We further demonstrate CausalFusion's multimodal capabilities through a joint image generation and captioning model, and showcase CausalFusion's ability for zero-shot in-context image manipulations. We hope that this work could provide the community with a fresh perspective on training multimodal models over discrete and continuous data.