mlsanityData to 14 Oct 2026

Next Semantic Scale Prediction via Hierarchical Diffusion Language Models

arXiv 2510.08632 ↗First posted 8 Oct 2025cs.CL cs.LGcited by 8

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

In this paper we introduce Hierarchical Diffusion Language Models (HDLM) -- a novel family of discrete diffusion models for language modeling. HDLM builds on a hierarchical vocabulary where low-level tokens with detailed semantics are surjectively mapped to high-level tokens with coarse-grained meanings. In the forward process, each token is independently perturbed to its higher-level ancestor with more abstract semantics according to the scheduler, while in the reverse process the model progressively predicts the next, more detailed semantics. Taken together, HDLM provides a general time-varying next semantic scale prediction process for language modeling. We derive closed-form expressions for the diffusion Evidence Lower Bound (ELBO), and show that HDLM can be implemented in a flexible manner while including the existing MDLM as a special case. We also propose practical training techniques based on the insights. Extensive text generation experiments validate the effectiveness of HDLM, which demonstrates consistently lower validation and generative perplexity than baselines.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 36 of its 42 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

36 of 42 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Dream 7B: Diffusion Large Language Models21 Aug
  2. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  3. 2025The Diffusion Duality12 Jun
  4. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  5. 2025Generalized Interpolating Discrete Diffusion6 Mar
  6. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  7. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  8. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  9. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  10. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  11. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  12. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
12 more
  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  2. 2017Deep Setscited by 1,421
  3. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  4. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  5. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  6. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  7. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 304
  8. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  9. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 283
  10. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  11. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 155
  12. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Modelscited by 6
  2. 2026Lattice Deduction Transformerscited by 3
  3. 2026Plan, Verify and Fill: A Structured Parallel Decoding Approach for Diffusion Language Modelscited by 2
  4. 2026Rethinking Token Prediction: Tree-Structured Diffusion Language Modelcited by 1
  5. 2026Multi-Mask Diffusion Language Models for Few-Step Generationcited by 1
  6. 2026Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attentioncited by 0
  7. 2026Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMscited by 0
  8. 2026Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling
  2. 2026Diffusion Language Models: An Experimental Analysis
  3. 2026Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design
  4. 2026dLLM: Simple Diffusion Language Modeling
  5. 2025Sequential Diffusion Language Models
  6. 2023Likelihood-Based Diffusion Language Models
  7. 2026Hierarchical Continuous Diffusion Language Models
  8. 2026DIP: Dynamic In-Context Planner For Diffusion Language Models