mlsanityData to 14 Oct 2026

Alpha Diffusion Language Models: Factorization Alone Is Not the Problem

arXiv 2609.38066 ↗First posted 29 Sep 2026cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Discrete diffusion language models can generate multiple tokens in parallel, but reducing the number of denoising steps can lead to inconsistent predictions. Standard cross-entropy training fits conditional token marginals, whereas parallel generation requires consistent joint predictions. We introduce Alpha Diffusion Language Models (AlphaDLM), trained with a sequence-level alpha loss that recovers cross-entropy in the limit of vanishing alpha and has a joint-mode optimum at alpha one. Our analysis characterizes how the objective and factorization jointly determine the fitted distribution. We identify conditions under which intermediate alpha preserves multiple valid completions while excluding invalid token combinations. Trained on TinyGSM, our method achieves 34.6% accuracy on GSM8K with only four model evaluations. We further scale the method to SDAR-1.7B and evaluate it on code and mathematics benchmarks. These results show that changing the training objective can improve the accuracy-computation trade-off of factorized diffusion language models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 33 of its 37 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

33 of 37 references.

Closest ancestors · 16

Same direction, newest first

  1. 2026Discrete Beckmann Transport Models for One-Step Language Modeling and Reasoning14 Sep
  2. 2026Conditional Total Correlation and the Serial Depth of Adaptive Parallel Sampling26 Aug
  3. 2026Language Modeling with Hyperspherical Flows11 May
  4. 2026Adaptation to Intrinsic Dependence in Diffusion Language Models23 Feb
  5. 2026IDLM: Inverse-distilled Diffusion Language Models22 Feb
  6. 2025CDLM: Consistency Diffusion Language Models For Faster Sampling24 Nov
  7. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation7 Oct
  8. 2025The Diffusion Duality12 Jun
  9. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  10. 2025Large Language Diffusion Models14 Feb
  11. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  12. 2024Discrete Copula Diffusion2 Oct
4 more
  1. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  2. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  4. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  6. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  7. 2017Non-Autoregressive Neural Machine Translationcited by 463
  8. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  9. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  10. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  11. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 60
  12. 2022On the Learning of Non-Autoregressive Transformerscited by 27
5 more
  1. 2019A Tunable Loss Function for Robust Classification: Calibration, Landscape, and Generalizationcited by 23
  2. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  3. 2026Posterior Refinement: Fast Language Generation via Any-Order Flow Mapscited by 11
  4. 2025Enabling Approximate Joint Sampling in Diffusion LMscited by 10
  5. 2026Tensor-Train Joint Modeling for Few-Step Discrete Diffusioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Factorization-Error-Free Discrete Diffusion Language Model via Speculative Decoding
  2. 2026Representation-Space MMD for Diffusion Language Models
  3. 2026Diffusion Language Models: An Experimental Analysis
  4. 2026Rethinking Token Prediction: Tree-Structured Diffusion Language Model
  5. 2026MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models
  6. 2026Scaling Beyond Masked Diffusion Language Models
  7. 2026Breaking the Factorization Barrier in Diffusion Language Models
  8. 2025Dream 7B: Diffusion Large Language Models