mlsanityData to 7 Oct 2026

Improved Large Language Diffusion Models

arXiv 2606.25331 ↗First posted 24 Jun 2026cs.CL cs.AI cs.LGcited by 12

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present iLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 43 of its 46 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

43 of 46 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models19 Mar
  2. 2025Diffusion Language Models are Super Data Learners5 Nov
  3. 2025Dream 7B: Diffusion Large Language Models21 Aug
  4. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models18 Aug
  5. 2025Diffusion Beats Autoregressive in Data-Constrained Settings21 Jul
  6. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  7. 2025Large Language Diffusion Models14 Feb
  8. 2024Scaling up Masked Diffusion Models on Text24 Oct
  9. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
  10. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  11. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  12. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
19 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2019Root Mean Square Layer Normalizationcited by 1,425
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  9. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  10. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  11. 2024Are We Done with MMLU?cited by 178
  12. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  13. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  14. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  15. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  16. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  17. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 30
  18. 2025Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Waycited by 22
  19. 2026LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Modelcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Don't Commit Alone: Joint Token Commitment in Diffusion Language Modelscited by 2
  2. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  3. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Modelscited by 1
  4. 2026Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Modelscited by 1
  5. 2026Affix Cache for Diffusion Large Language Modelscited by 0
  6. 2026A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Modelscited by 0
  7. 2026dQwen3.5: Hybrid-Attention Diffusion Language Modelscited by 0
  8. 2026Circuit Hypernetworks for Quantum-Augmented Diffusion Language Modelscited by 0
  9. 2026On Trajectory-Aware Training for Masked Diffusion Language Modelscited by 0
  10. 2026How Should Diffusion Language Models Edit Code?cited by 0
  11. 2026Does This Action Still Explain the Task? Reverse Scoring for Diffusion Language Model Agentscited by 0
  12. 2026Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding and Accelerating the Training of Masked Diffusion Language Models
  2. 2026DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
  3. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
  4. 2026Diffusion Language Models: An Experimental Analysis
  5. 2026Fine-Tuning Diffusion Language Models with Context Selection and Target Weighting
  6. 2025Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning
  7. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100B
  8. 2025Masked Diffusion Language Models with Frequency-Informed Training