mlsanityData to 7 Oct 2026

SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation

arXiv 2510.06303 ↗First posted 7 Oct 2025cs.LG cs.AIcited by 118

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

We propose SDAR, a Synergistic Diffusion-Autoregression paradigm that unifies the training efficiency of autoregressive models with the parallel inference capability of diffusion. Instead of costly end-to-end diffusion training, SDAR performs a lightweight paradigm conversion that transforms a well-trained autoregressive (AR) model into a blockwise diffusion model through brief, data-efficient adaptation. During inference, SDAR generates sequences autoregressively across blocks for global coherence while decoding all tokens within each block in parallel via a discrete diffusion process. Extensive experiments show that AR models remain substantially more compute-efficient than masked diffusion models, providing a strong foundation for adaptation. Building on this insight, SDAR achieves efficient AR-to-diffusion conversion with minimal cost, preserving AR-level performance while enabling parallel generation. Scaling studies across dense and Mixture-of-Experts architectures confirm that SDAR scales without compromise: larger models exhibit stronger robustness to block size and decoding thresholds, yielding greater speedups without accuracy loss. Beyond efficiency, SDAR demonstrates enhanced reasoning and domain adaptability. Our 30B MoE model surpasses its AR counterpart on challenging scientific reasoning benchmarks such as GPQA and ChemBench, and gains further improvements under test-time scaling methods like majority voting and pass@k. Together, these results establish SDAR as a practical paradigm that combines the strengths of autoregression and diffusion for scalable, high-throughput reasoning.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 50 of its 61 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

50 of 61 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Dream 7B: Diffusion Large Language Models21 Aug
  2. 2025A Survey on Diffusion Language Models14 Aug
  3. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  4. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  5. 2025Discrete Diffusion in Large Language and Multimodal Models: A Survey16 Jun
  6. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  7. 2025Large Language Diffusion Models14 Feb
  8. 2025Theoretical Benefit and Limitation of Diffusion Language Model13 Feb
  9. 2024Scaling up Masked Diffusion Models on Text24 Oct
  10. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  11. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
  12. 2024Simple and Effective Masked Diffusion Language Models11 Jun
5 more
  1. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  3. 2023Likelihood-Based Diffusion Language Models30 May
  4. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  5. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  12. 2023Let's Verify Step by Stepcited by 3,598
21 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2023Large Language Modelscited by 774
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  11. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  12. 2022ProGen2: Exploring the Boundaries of Protein Language Modelscited by 156
  13. 2024The pitfalls of next-token predictioncited by 156
  14. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 103
  15. 2023Amortizing intractable inference in large language modelscited by 94
  16. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  17. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 59
  18. 2025PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solvingcited by 31
  19. 2024A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understandingcited by 19
  20. 2025Unifying Autoregressive and Diffusion-Based Sequence Generationcited by 17
  21. 2025From Sentences to Sequences: Rethinking Languages in Biological Systemcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 104.

118 papers.

Sort:

By area:

  1. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  2. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  3. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 38
  4. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  5. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 30
  6. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modelingcited by 28
  7. 2025Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Waycited by 22
  8. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMscited by 22
  9. 2025CDLM: Consistency Diffusion Language Models For Faster Samplingcited by 20
  10. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 19
  11. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  12. 2025LoPA: Scaling dLLM Inference via Lookahead Parallel Decodingcited by 17
38 more
  1. 2025AsyncVLA: Asynchronous Flow Matching for Vision-Language-Action Modelscited by 16
  2. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 14
  3. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 14
  4. 2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
  5. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 12
  6. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 11
  7. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  8. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Modelscited by 10
  9. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  10. 2026MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generationcited by 10
  11. 2026Residual Context Diffusion Language Modelscited by 9
  12. 2026Stabilizing Reinforcement Learning for Diffusion Language Modelscited by 9
  13. 2026Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrowcited by 8
  14. 2026TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Accelerationcited by 8
  15. 2026LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Modelcited by 8
  16. 2026ρ-textttEOS: Training-free Bidirectional Variable-Length Control for Masked Diffusion LLMscited by 7
  17. 2026DLLM-Searcher: Adapting Diffusion Large Language Model for Search Agentscited by 7
  18. 2026MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decodingcited by 7
  19. 2026When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Modelscited by 7
  20. 2025Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Modelscited by 6
  21. 2026Improving Sampling for Masked Diffusion Models via Information Gaincited by 6
  22. 2026LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learningcited by 6
  23. 2026Introspective Diffusion Language Modelscited by 6
  24. 2026T^star: Progressive Block Scaling for Masked Diffusion Language Models Through Trajectory Aware Reinforcement Learningcited by 5
  25. 2026S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculationcited by 5
  26. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executorcited by 5
  27. 2026A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Modelscited by 5
  28. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMscited by 4
  29. 2026Learning from the Self-future: On-policy Self-distillation for dLLMscited by 4
  30. 2026DreamReasoner-8B: Block-Size Curriculum Learning for Diffusion Reasoning Modelscited by 4
  31. 2026Trace-Based On-Policy Distillation for Masked Diffusion Language Modelscited by 4
  32. 2026Causal Autoregressive Diffusion Language Modelcited by 3
  33. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  34. 2026FOCUS: DLLMs Know How to Tame Their Compute Boundcited by 3
  35. 2026Few-Step Diffusion Language Models via Trajectory Self-Distillationcited by 3
  36. 2026MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Modelscited by 3
  37. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3
  38. 2026Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusioncited by 3

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TiDAR: Think in Diffusion, Talk in Autoregression
  2. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  3. 2026Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks
  4. 2025Sequential Diffusion Language Models
  5. 2025DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
  6. 2026Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference
  7. 2025On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond
  8. 2025Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation