mlsanityData to 14 Oct 2026

Gumbel Distillation for Parallel Text Generation

arXiv 2603.22216 ↗First posted 23 Mar 2026cs.CL cs.LGcited by 1

On the map: Efficient LLM inference and serving → Diffusion LLM and MoE inference

Abstract (arXiv)

The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods. However, these non-AR models often sacrifice generation quality as they struggle to model the complex joint distribution of token sequences. To narrow this performance gap, we introduce Gumbel Distillation, a novel distillation technique that enables parallel decoders to learn this distribution effectively. Our method leverages the Gumbel-Max trick to create a deterministic mapping from a latent Gumbel noise space to the output tokens of a high-performing AR teacher. As a model-agnostic technique, Gumbel Distillation seamlessly integrates with diverse parallel decoding architectures, including MDLM and BD3-LM. Experiments on LM1B and OpenWebText show that Gumbel Distillation substantially improves the generation quality of parallel language models, achieving a 30.0% improvement in MAUVE score and 10.5% in generative perplexity over MDLM trained on OpenWebText dataset. Code available at https://github.com/hxixixh/gumbel-distill.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Diffusion LLM and MoE inference, one of 12 directions in Efficient LLM inference and serving. Builds on 72 of its 82 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

72 of 82 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decoding31 May
  2. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding28 May

Other references · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Flow Matching for Generative Modelingcited by 5,702
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
38 more
  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  13. 2025Large Language Diffusion Modelscited by 912
  14. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  15. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  16. 2016Sequence-Level Knowledge Distillationcited by 801
  17. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  18. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  19. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  20. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  21. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  22. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  23. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  24. 2017Non-Autoregressive Neural Machine Translationcited by 463
  25. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  26. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 450
  27. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  28. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  29. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  30. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  31. 2024Discrete Flow Matchingcited by 317
  32. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  33. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  34. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  35. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  36. 2022Continuous diffusion for categorical datacited by 214
  37. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 211
  38. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Gumbel Straight Flow: Distilling Autoregressive Models into One-step Flow Mapscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models
  2. 2026Locally Coherent Parallel Decoding in Diffusion Language Models
  3. 2024Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
  4. 2025dParallel: Learnable Parallel Decoding for dLLMs
  5. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  6. 2023PaSS: Parallel Speculative Sampling
  7. 2022A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
  8. 2026Ripple-Pivot Search: Active Parallel Decoding for Diffusion Large Language Models