# Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

arXiv: https://arxiv.org/abs/2607.05722 · HTML: https://mlsanity.com/paper/2607.05722/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 65 references that are in the map.
- First posted on arXiv 7 Jul 2026.

## Builds on (most cited first, 30 of 61)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Dream 7B: Diffusion Large Language Models](https://mlsanity.com/paper/2508.15487/) (2025)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding](https://mlsanity.com/paper/2505.22618/) (2025)
- [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://mlsanity.com/paper/2503.09573/) (2025)
- [Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data](https://mlsanity.com/paper/2406.03736/) (2024)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models](https://mlsanity.com/paper/2505.19223/) (2025)
- [Scaling Diffusion Language Models via Adaptation from Autoregressive Models](https://mlsanity.com/paper/2410.17891/) (2024)
- [MMaDA: Multimodal Large Diffusion Language Models](https://mlsanity.com/paper/2505.15809/) (2025)
- [Mercury: Ultra-Fast Language Models Based on Diffusion](https://mlsanity.com/paper/2506.17298/) (2025)
- [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022)
- [Scaling up Masked Diffusion Models on Text](https://mlsanity.com/paper/2410.18514/) (2024)
- [SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control](https://mlsanity.com/paper/2210.17432/) (2022)

## Built on by (most cited first, 24 of 24)

- [xPress: Parallel Refinement for Diffusion Drafters in Speculative Decoding](https://mlsanity.com/paper/2608.02438/) (2026)
- [LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding](https://mlsanity.com/paper/2608.20530/) (2026)
- [Unlocking Lossless Speedups in LLMs via Discrete Diffusion](https://mlsanity.com/paper/2609.04010/) (2026)
- [Recommendation Retrievers Need Verifiers: Universal Generative Reranking for Sequential Recommendations](https://mlsanity.com/paper/2609.12270/) (2026)
- [Reducing Pretraining-Generation Mismatch in Diffusion Language Models](https://mlsanity.com/paper/2608.09424/) (2026)
- [Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosis](https://mlsanity.com/paper/2608.20743/) (2026)
- [Prefix-Denoising Consistency: Test-Time Verification for Diffusion Language Models](https://mlsanity.com/paper/2608.25311/) (2026)
- [Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training](https://mlsanity.com/paper/2609.19242/) (2026)
- [dQwen3.5: Hybrid-Attention Diffusion Language Models](https://mlsanity.com/paper/2609.20751/) (2026)
- [Diffusion Drafts, AR Verifies: Accelerating Document OCR with Self-Speculative Decoding](https://mlsanity.com/paper/2609.26638/) (2026)
- [Reciprocal Guidance: Orchestrating Draft and Verify Budgets for Advancing the Diffusion-AR Self-Speculation Frontier](https://mlsanity.com/paper/2609.34388/) (2026)
- [One Readout, Many Repairs: Diffusion-Guided Hierarchical Search for Tool-Agent Repair](https://mlsanity.com/paper/2609.34879/) (2026)
- [HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks](https://mlsanity.com/paper/2609.36357/) (2026)
- [Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models](https://mlsanity.com/paper/2610.00894/) (2026)
- [Two Clocks in Diffusion MLLMs: When Answers Stabilize Before Rationales Unfold](https://mlsanity.com/paper/2610.00953/) (2026)
- [ALoDLM: Adaptively Looped Diffusion Language Models](https://mlsanity.com/paper/2610.04198/) (2026)
- [SpecFold: Folding Multi-Branch Redundancy for Faster Speculative Decoding in Diffusion Language Models](https://mlsanity.com/paper/2610.04875/) (2026)
- [Representation-Space MMD for Diffusion Language Models](https://mlsanity.com/paper/2610.06648/) (2026)
- [SoloQ: Calibration-Free Quantization for Diffusion Language Models](https://mlsanity.com/paper/2610.07121/) (2026)
- [Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution](https://mlsanity.com/paper/2610.07822/) (2026)
- [DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusion](https://mlsanity.com/paper/2610.12214/) (2026)
- [Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models](https://mlsanity.com/paper/2610.17590/) (2026)
- [Full-duplex Spoken Language Modeling with Discrete Diffusion](https://mlsanity.com/paper/2610.17780/) (2026)
- [One Block Ahead: Promoting Sampler State in Diffusion Language Models](https://mlsanity.com/paper/2610.19510/) (2026)

## Abstract (arXiv)

We introduce Nemotron-Labs-Diffusion, a tri-mode language model (LM) that unifies AR, diffusion, and self-speculation decoding within a single architecture. Trained with a joint AR-diffusion objective, Nemotron-Labs-Diffusion can switch modes to sustain high throughput across deployment settings and concurrency levels. Our study shows that (1) AR and diffusion objectives are complementary: diffusion improves lookahead planning, while AR provides left-to-right linguistic priors. (2) In self-speculation mode, diffusion drafts while AR verifies, outperforming multi-token prediction (MTP) methods in both acceptance rate and real-device efficiency. (3) A speed-of-light analysis further demonstrates diffusion's long-term potential, with up to 76.5% more tokens per forward pass than self-speculation under an optimal sampler. Scaling to 3B, 8B, and 14B parameters, our Nemotron-Labs-Diffusion family, including base, instruct, and vision-language models, consistently outperforms state-of-the-art open-source AR and diffusion LMs in both accuracy and speed. For example, Nemotron-Labs-Diffusion-8B decodes 6x more tokens per forward than Qwen3-8B with comparable accuracy, translating to 4x higher throughput on SPEED-Bench with SGLang on a GB200 GPU.