mlsanityData to 7 Oct 2026

Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

arXiv 2607.05722 ↗First posted 7 Jul 2026cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

We introduce Nemotron-Labs-Diffusion, a tri-mode language model (LM) that unifies AR, diffusion, and self-speculation decoding within a single architecture. Trained with a joint AR-diffusion objective, Nemotron-Labs-Diffusion can switch modes to sustain high throughput across deployment settings and concurrency levels. Our study shows that (1) AR and diffusion objectives are complementary: diffusion improves lookahead planning, while AR provides left-to-right linguistic priors. (2) In self-speculation mode, diffusion drafts while AR verifies, outperforming multi-token prediction (MTP) methods in both acceptance rate and real-device efficiency. (3) A speed-of-light analysis further demonstrates diffusion's long-term potential, with up to 76.5% more tokens per forward pass than self-speculation under an optimal sampler. Scaling to 3B, 8B, and 14B parameters, our Nemotron-Labs-Diffusion family, including base, instruct, and vision-language models, consistently outperforms state-of-the-art open-source AR and diffusion LMs in both accuracy and speed. For example, Nemotron-Labs-Diffusion-8B decodes 6x more tokens per forward than Qwen3-8B with comparable accuracy, translating to 4x higher throughput on SPEED-Bench with SGLang on a GB200 GPU.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 61 of its 65 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

61 of 65 references.

References · 61

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  10. 2025Large Language Diffusion Modelscited by 872
  11. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  12. 2016A Diagram Is Worth A Dozen Imagescited by 793
38 more
  1. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  2. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  4. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  5. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  7. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 439
  8. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  9. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  10. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  11. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  12. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  13. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  14. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  15. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  16. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  17. 2024Scaling up Masked Diffusion Models on Textcited by 195
  18. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Controlcited by 191
  19. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  20. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  21. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  22. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  23. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  24. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  25. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  26. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  27. 2025The Diffusion Dualitycited by 124
  28. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  29. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcingcited by 111
  30. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  31. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  32. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  33. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  34. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  35. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  36. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  37. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  38. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 42

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 19.

19 papers.

Sort:

By area:

  1. 2026xPress: Parallel Refinement for Diffusion Drafters in Speculative Decodingcited by 7
  2. 2026LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decodingcited by 2
  3. 2026Unlocking Lossless Speedups in LLMs via Discrete Diffusioncited by 1
  4. 2026Recommendation Retrievers Need Verifiers: Universal Generative Reranking for Sequential Recommendationscited by 1
  5. 2026Reducing Pretraining-Generation Mismatch in Diffusion Language Modelscited by 0
  6. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0
  7. 2026Prefix-Denoising Consistency: Test-Time Verification for Diffusion Language Modelscited by 0
  8. 2026Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Trainingcited by 0
  9. 2026dQwen3.5: Hybrid-Attention Diffusion Language Modelscited by 0
  10. 2026Diffusion Drafts, AR Verifies: Accelerating Document OCR with Self-Speculative Decodingcited by 0
  11. 2026Reciprocal Guidance: Orchestrating Draft and Verify Budgets for Advancing the Diffusion-AR Self-Speculation Frontiercited by 0
  12. 2026One Readout, Many Repairs: Diffusion-Guided Hierarchical Search for Tool-Agent Repaircited by 0
7 more
  1. 2026HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworkscited by 0
  2. 2026Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Modelscited by 0
  3. 2026Two Clocks in Diffusion MLLMs: When Answers Stabilize Before Rationales Unfoldcited by 0
  4. 2026SpecFold: Folding Multi-Branch Redundancy for Faster Speculative Decoding in Diffusion Language Modelscited by 0
  5. 2026SoloQ: Calibration-Free Quantization for Diffusion Language Modelscited by 0
  6. 2026Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distributioncited by 0
  7. 2026DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context
  2. 2026Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?
  3. 2026Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks
  4. 2026S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation
  5. 2025How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
  6. 2026SimSD: Simple Speculative Decoding in Diffusion Language Models
  7. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
  8. 2026Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference