# Simplex Diffusion Models

arXiv: https://arxiv.org/abs/2609.35553 · HTML: https://mlsanity.com/paper/2609.35553/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 36 of its 140 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [Continuous diffusion for categorical data](https://mlsanity.com/paper/2211.15089/) (2022)
- [Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](https://mlsanity.com/paper/2206.02369/) (2022)
- [TinyGSM: achieving >80% on GSM8k with small language models](https://mlsanity.com/paper/2312.09241/) (2023)
- [Flow Map Language Models: One-step Language Modeling via Continuous Denoising](https://mlsanity.com/paper/2602.16813/) (2026)
- [Mixed Continuous and Categorical Flow Matching for 3D De Novo Molecule Generation](https://mlsanity.com/paper/2404.19739/) (2024)
- [ELF: Embedded Language Flows](https://mlsanity.com/paper/2605.10938/) (2026)
- [Categorical SDEs with Simplex Diffusion](https://mlsanity.com/paper/2210.14784/) (2022)
- [LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling](https://mlsanity.com/paper/2604.11748/) (2026)
- [Discrete Flow Maps](https://mlsanity.com/paper/2604.09784/) (2026)
- [Language Modeling with Hyperspherical Flows](https://mlsanity.com/paper/2605.11125/) (2026)
- [Continuous Diffusion Scales Competitively with Discrete Diffusion for Language](https://mlsanity.com/paper/2605.18530/) (2026)
- [α-Flow: A Unified Framework for Continuous-State Discrete Flow Matching Models](https://mlsanity.com/paper/2504.10283/) (2025)
- [Generative Frontiers: Why Evaluation Matters for Diffusion Language Models](https://mlsanity.com/paper/2604.02718/) (2026)
- [CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think](https://mlsanity.com/paper/2603.02547/) (2026)
- [Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation](https://mlsanity.com/paper/2605.22765/) (2026)
- [Promises, Outlooks and Challenges of Diffusion Language Modeling](https://mlsanity.com/paper/2406.11473/) (2024)
- [Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD](https://mlsanity.com/paper/2603.20155/) (2026)
- [DiffusionGemma Technical Report](https://mlsanity.com/paper/2608.00146/) (2026)
- [CoBit: Language Modeling with Bitstream Diffusion](https://mlsanity.com/paper/2605.07013/) (2026)
- [Generative Modeling of Discrete Joint Distributions by E-Geodesic Flow Matching on Assignment Manifolds](https://mlsanity.com/paper/2402.07846/) (2024)

## Built on by (most cited first, 1 of 1)

- [Scaling and Distilling Text Embeddings for Better Diffusibility](https://mlsanity.com/paper/2610.01016/) (2026)

## Abstract (arXiv)

Diffusion models have revolutionized generative modeling for continuous data through the gradual refinement of a belief state. This iterative refinement has not yet carried over to discrete diffusion models, which discard uncertainty at intermediate steps through categorical sampling (information collapse). We propose Simplex Diffusion Models (SDMs), a framework that lifts the diffusion process to the probability simplex to represent beliefs over categories. SDMs admit probability paths with closed-form reverse transitions and can be trained with a simple cross-entropy loss. Contrary to earlier proposals such as Dirichlet Flow Matching which requires integrating an ordinary differential equation, we introduce a DDIM-like sampler with a tunable level of stochasticity. Because SDMs operate on samples on the simplex, they can carry uncertainty across denoising steps, which mitigates information collapse. On OpenWebText, SDMs are competitive with strong Discrete Diffusion baselines, achieving 17.0 GenPPL at 5.46 unigram entropy in 64 sampling steps, close to real validation data. Even without Self-Conditioning (SC), SDMs outperform masked and uniform diffusion (with SC or predictor-corrector sampling) on code generation (TinyGSM, T=0.1; 49.0% vs. 45.8%). Distilled down to 8 steps, SDMs solve 32.1% of GSM8K problems, more than distilled Discrete Diffusion models with 128 steps (21.4%).