# DiffusionGemma Technical Report

arXiv: https://arxiv.org/abs/2608.00146 · HTML: https://mlsanity.com/paper/2608.00146/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 33 of its 150 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 30 of 33)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [Gemma 4 Technical Report](https://mlsanity.com/paper/2607.02770/) (2026)
- [Mercury: Ultra-Fast Language Models Based on Diffusion](https://mlsanity.com/paper/2506.17298/) (2025)
- [Continuous diffusion for categorical data](https://mlsanity.com/paper/2211.15089/) (2022)
- [Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference](https://mlsanity.com/paper/2508.02193/) (2025)
- [LLaDA2.0: Scaling Up Diffusion Language Models to 100B](https://mlsanity.com/paper/2512.15745/) (2025)
- [Fast-dLLM v2: Efficient Block-Diffusion LLM](https://mlsanity.com/paper/2509.26328/) (2025)
- [Self-conditioned Embedding Diffusion for Text Generation](https://mlsanity.com/paper/2211.04236/) (2022)
- [CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings](https://mlsanity.com/paper/2501.01257/) (2025)
- [Flow Map Language Models: One-step Language Modeling via Continuous Denoising](https://mlsanity.com/paper/2602.16813/) (2026)
- [EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Sampling](https://mlsanity.com/paper/2403.14541/) (2024)
- [ELF: Embedded Language Flows](https://mlsanity.com/paper/2605.10938/) (2026)
- [An Introduction to Flow Matching and Diffusion Models](https://mlsanity.com/paper/2506.02070/) (2025)
- [Discrete Flow Maps](https://mlsanity.com/paper/2604.09784/) (2026)
- [Language Modeling with Hyperspherical Flows](https://mlsanity.com/paper/2605.11125/) (2026)
- [Promises, Outlooks and Challenges of Diffusion Language Modeling](https://mlsanity.com/paper/2406.11473/) (2024)
- [Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD](https://mlsanity.com/paper/2603.20155/) (2026)
- [Reverse That Number! Decoding Order Matters in Arithmetic Learning](https://mlsanity.com/paper/2403.05845/) (2024)
- [Transfer Learning for Text Diffusion Models](https://mlsanity.com/paper/2401.17181/) (2024)
- [Accelerating Discrete Diffusion Models with Parallel-In-Time Sampling](https://mlsanity.com/paper/2607.00773/) (2026)

## Built on by (most cited first, 11 of 11)

- [Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models](https://mlsanity.com/paper/2609.15177/) (2026)
- [Parallelism, critical windows, and separations among diffusion language models](https://mlsanity.com/paper/2609.20539/) (2026)
- [Simplex Diffusion Models](https://mlsanity.com/paper/2609.35553/) (2026)
- [On Trajectory-Aware Training for Masked Diffusion Language Models](https://mlsanity.com/paper/2609.37974/) (2026)
- [UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding](https://mlsanity.com/paper/2609.39972/) (2026)
- [Scaling and Distilling Text Embeddings for Better Diffusibility](https://mlsanity.com/paper/2610.01016/) (2026)
- [ALoDLM: Adaptively Looped Diffusion Language Models](https://mlsanity.com/paper/2610.04198/) (2026)
- [Representation-Space MMD for Diffusion Language Models](https://mlsanity.com/paper/2610.06648/) (2026)
- [SoloQ: Calibration-Free Quantization for Diffusion Language Models](https://mlsanity.com/paper/2610.07121/) (2026)
- [Enhancing Diffusion Language Models with Autoregressive Post-Training Weights](https://mlsanity.com/paper/2610.08108/) (2026)
- [Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation](https://mlsanity.com/paper/2610.14162/) (2026)

## Abstract (arXiv)

We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.