# CodeFusion: A Pre-trained Diffusion Model for Code Generation

arXiv: https://arxiv.org/abs/2310.17680 · HTML: https://mlsanity.com/paper/2310.17680/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://mlsanity.com/paper/1805.08949/) (2018)
- [AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation](https://mlsanity.com/paper/2305.09515/) (2023)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System](https://mlsanity.com/paper/1802.08979/) (2018)
- [Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise](https://mlsanity.com/paper/2212.11685/) (2022)
- [Non-Autoregressive Text Generation with Pre-trained Language Models](https://mlsanity.com/paper/2102.08220/) (2021)
- [An Empirical Comparison of Pre-Trained Models of Source Code](https://mlsanity.com/paper/2302.04026/) (2023)
- [Toward Code Generation: A Survey and Lessons from Semantic Parsing](https://mlsanity.com/paper/2105.03317/) (2021)
- [TransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skills](https://mlsanity.com/paper/2306.07285/) (2023)

## Built on by (most cited first, 30 of 31)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation](https://mlsanity.com/paper/2506.20639/) (2025)
- [LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs](https://mlsanity.com/paper/2312.04372/) (2023)
- [Logits of API-Protected LLMs Leak Proprietary Information](https://mlsanity.com/paper/2403.09539/) (2024)
- [LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning](https://mlsanity.com/paper/2510.04573/) (2025)
- [From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models](https://mlsanity.com/paper/2511.21103/) (2025)
- [Proving the Coding Interview: A Benchmark for Formally Verified Code Generation](https://mlsanity.com/paper/2502.05714/) (2025)
- [DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas](https://mlsanity.com/paper/2602.01326/) (2026)
- [Exploring the Potential of Diffusion Large Language Models in Code Generation](https://mlsanity.com/paper/2509.11252/) (2025)
- [TreeDiff: AST-Guided Code Generation with Diffusion LLMs](https://mlsanity.com/paper/2508.01473/) (2025)
- [Pragmatic Competence Evaluation of Large Language Models for the Korean Language](https://mlsanity.com/paper/2403.12675/) (2024)
- [FALCON: Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization system](https://mlsanity.com/paper/2410.21349/) (2024)
- [MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?](https://mlsanity.com/paper/2507.19598/) (2025)
- [Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks](https://mlsanity.com/paper/2504.19444/) (2025)
- [Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints](https://mlsanity.com/paper/2501.08246/) (2025)
- [Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?](https://mlsanity.com/paper/2604.02028/) (2026)
- [A Survey on Large Language Models from Concept to Implementation](https://mlsanity.com/paper/2403.18969/) (2024)
- [A Neurosymbolic Agent System for Compositional Visual Reasoning](https://mlsanity.com/paper/2506.07778/) (2025)
- [A Novel Diffusion Model for Pairwise Geoscience Data Generation with Unbalanced Training Dataset](https://mlsanity.com/paper/2501.00941/) (2025)
- [System Log Parsing with Large Language Models: A Review](https://mlsanity.com/paper/2504.04877/) (2025)
- [Diffusion is a code repair operator and generator](https://mlsanity.com/paper/2508.11110/) (2025)
- [DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels](https://mlsanity.com/paper/2602.11715/) (2026)
- [Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One](https://mlsanity.com/paper/2609.04531/) (2026)
- [MoE-DiffuSeq: Enhancing Long-Document Diffusion Models with Sparse Attention and Mixture of Experts](https://mlsanity.com/paper/2512.20604/) (2025)
- [Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation](https://mlsanity.com/paper/2602.01187/) (2026)
- [A Survey on the Real Power of ChatGPT](https://mlsanity.com/paper/2405.00704/) (2024)
- [Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?](https://mlsanity.com/paper/2409.09568/) (2024)
- [Eliciting Instruction-tuned Code Language Models' Capabilities to Utilize Auxiliary Function for Code Generation](https://mlsanity.com/paper/2409.13928/) (2024)
- [Tabularis Formatus: Predictive Formatting for Tables](https://mlsanity.com/paper/2508.11121/) (2025)
- [Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code](https://mlsanity.com/paper/2606.23690/) (2026)

## Abstract (arXiv)

Imagine a developer who can only change their last line of code, how often would they have to start writing a function from scratch before it is correct? Auto-regressive models for code generation from natural language have a similar limitation: they do not easily allow reconsidering earlier tokens generated. We introduce CodeFusion, a pre-trained diffusion code generation model that addresses this limitation by iteratively denoising a complete program conditioned on the encoded natural language. We evaluate CodeFusion on the task of natural language to code generation for Bash, Python, and Microsoft Excel conditional formatting (CF) rules. Experiments show that CodeFusion (75M parameters) performs on par with state-of-the-art auto-regressive systems (350M-175B parameters) in top-1 accuracy and outperforms them in top-3 and top-5 accuracy due to its better balance in diversity versus quality.