mlsanityData to 14 Oct 2026

Esoteric Language Models: A Family of Any-Order Diffusion LLMs

arXiv 2506.01928 ↗First posted 2 Jun 2025cs.CL cs.LGcited by 34

Not on the map. Why?

Cited by 29 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Diffusion-based language models offer a compelling alternative to autoregressive (AR) models by enabling parallel and controllable generation. Within this family, Masked Diffusion Models (MDMs) currently perform best but still underperform AR models in perplexity and lack key inference-time efficiency features, most notably KV caching. We introduce Eso-LMs, a new family of models that fuses AR and MDM paradigms, smoothly interpolating between their perplexities while overcoming their respective limitations. Unlike prior work, which uses transformers with bidirectional attention as MDM denoisers, we exploit the connection between MDMs and Any-Order autoregressive models and adopt causal attention. This design lets us compute the exact likelihood of MDMs for the first time and, crucially, enables us to introduce KV caching for MDMs while preserving parallel generation for the first time, significantly improving inference efficiency. Combined with an optimized sampling schedule, Eso-LMs establish a new state of the art on the speed-quality Pareto frontier for unconditional generation. We provide the code, model checkpoints, and the video tutorial on the project page: https://s-sahoo.com/Eso-LMs.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 49 of its 63 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

49 of 63 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
37 more
  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  2. 2021Variational Diffusion Modelscited by 1,153
  3. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  5. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  6. 2025Large Language Diffusion Modelscited by 912
  7. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  8. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  9. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  10. 2022Efficiently Scaling Transformer Inferencecited by 539
  11. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  12. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  13. 2015Importance Weighted Autoencoderscited by 456
  14. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  15. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  16. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  17. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  18. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  19. 2021Autoregressive Diffusion Modelscited by 210
  20. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  21. 2025Remasking Discrete Diffusion Models with Inference-Time Scalingcited by 189
  22. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  23. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  24. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 155
  25. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 143
  26. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 129
  27. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 113
  28. 2023Generative Diffusion Models on Graphs: Methods and Applicationscited by 81
  29. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  30. 2013A Deep and Tractable Density Estimatorcited by 68
  31. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  32. 2024Discrete Copula Diffusioncited by 54
  33. 2023Diffusion Models With Learned Adaptive Noisecited by 51
  34. 2025GenMol: A Drug Discovery Generalist with Discrete Diffusioncited by 50
  35. 2023Information-Theoretic Diffusioncited by 44
  36. 2024σ-GPTs: A New Approach to Autoregressive Modelscited by 20
  37. 2025Enabling Autoregressive Models to Fill In Masked Tokenscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 17.

34 papers.

Sort:

By area:

  1. 2025The Diffusion Dualitycited by 135
  2. 2025Any-Order Flexible Length Masked Diffusioncited by 51
  3. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 41
  4. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  5. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modelingcited by 36
  6. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 35
  7. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  8. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  9. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  10. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimationcited by 27
  11. 2026Scaling Beyond Masked Diffusion Language Modelscited by 27
  12. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
22 more
  1. 2025Set Block Decoding is a Language Model Inference Acceleratorcited by 23
  2. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  3. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 19
  4. 2026Continuous Diffusion Scales Competitively with Discrete Diffusion for Languagecited by 19
  5. 2025ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decodingcited by 18
  6. 2026The Diffusion Duality, Chapter II: Psi-Samplerscited by 12
  7. 2025Enabling Approximate Joint Sampling in Diffusion LMscited by 10
  8. 2025Planned Diffusioncited by 9
  9. 2026Reasoning with Latent Tokens in Diffusion Language Modelscited by 9
  10. 2026IDLM: Inverse-distilled Diffusion Language Modelscited by 8
  11. 2026Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processescited by 7
  12. 2025Rethinking Reasoning with MDLMs: Early Exits, Post-hoc Reasoning, and Beyondcited by 4
  13. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantagescited by 4
  14. 2025Masked Diffusion Models are Secretly Learned-Order Autoregressive Modelscited by 3
  15. 2026TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Modelscited by 2
  16. 2026Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decodingcited by 2
  17. 2026Can I Have Your Order? Monte-Carlo Tree Search for Slot Filling Ordering in Diffusion Language Modelscited by 1
  18. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Modelscited by 1
  19. 2026Scaling Inherently Interpretable Language Modelscited by 1
  20. 2026Why Any-Order Autoregressive Models Need Two-Stream Attention: A Structural-Semantic Tradeoffcited by 0
  21. 2026DualDiffusion: A Speculative Decoding Strategy for Masked Diffusion Modelscited by 0
  22. 2026Ambient Discrete Diffusion: Using the Wrong Data at the Right Time for Data Efficient Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Diffusion Language Models
  2. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
  3. 2025Dream 7B: Diffusion Large Language Models
  4. 2026Auto-Regressive Masked Diffusion Models
  5. 2025Diffusion Language Models are Super Data Learners
  6. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  7. 2025Diffusion Beats Autoregressive in Data-Constrained Settings
  8. 2025On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond