mlsanityData to 7 Oct 2026

Understanding Evaluation Illusion in Diffusion Large Language Models

arXiv 2606.29228 ↗First posted 28 Jun 2026cs.CL cs.LGcited by 0

On the map: Efficient LLM inference and serving → Diffusion LLM and MoE inference

Abstract (arXiv)

Despite the capability of parallel decoding, diffusion large language models (dLLMs) require many denoising steps to maintain generation quality, motivating recent research on efficient decoding strategies. However, existing studies have reported inconsistent evaluation results even under seemingly identical evaluation settings, risking biased conclusions about dLLM decoding methods. To understand this evaluation concern, we conduct a rigorous evaluation of current decoding methods for dLLMs across diverse evaluation settings. Surprisingly, our analysis reveals that the ranking of decoding methods is highly sensitive to the choice of prompt templates. Single-template evaluation can lead to an illusion that decoding methods improve inference efficiency without performance degradation. Through comprehensive experiments, we find that current parallel decoding methods consistently underperform the single-token decoding baseline, failing to overcome the speed-quality trade-off. We further identify this evaluation inconsistency as the high sensitivity of parallel decoding methods to minor variations in prompt templates. Our experiments show that an effective prompt template can achieve strong evaluation results even with fewer denoising steps, markedly outperforming the marginal gain from increasing denoising steps. Beyond prompt templates, our experiments indicate that overlooked evaluation settings can also notably affect the assessment of decoding methods. Based on these findings, we propose practical guidelines for the reliable evaluation of decoding methods in dLLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion LLM and MoE inference, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 44 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

40 of 44 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM27 May
  2. 2025Attention Is All You Need for KV Cache in Diffusion LLMs16 Oct
  3. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs6 Oct
  4. 2025Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models30 Sep
  5. 2025dParallel: Learnable Parallel Decoding for dLLMs30 Sep
  6. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding29 Sep
  7. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching27 Sep
  8. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction4 Aug
  9. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding28 May
  10. 2025dKV-Cache: The Cache for Diffusion Language Models21 May

Other references · 30

  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2025Large Language Diffusion Modelscited by 872
  6. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  7. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  8. 2021True Few-Shot Learning with Language Modelscited by 343
  9. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  10. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  11. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  12. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
18 more
  1. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  2. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  3. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  4. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  5. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  6. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  7. 2024Efficient multi-prompt evaluation of LLMscited by 72
  8. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 47
  9. 2025Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMscited by 45
  10. 2025Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principlescited by 43
  11. 2025Benchmarking Prompt Sensitivity in Large Language Modelscited by 38
  12. 2025DPad: Efficient Diffusion Language Models with Suffix Dropoutcited by 38
  13. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 29
  14. 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMscited by 23
  15. 2026Parallel-Probe: Towards Efficient Parallel Thinking via 2D Probingcited by 13
  16. 2024Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templatescited by 7
  17. 2026ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skippingcited by 6
  18. 2026Lookahead Path Likelihood Optimization for Diffusion LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers
  2. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
  3. 2025How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
  4. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  5. 2025Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models
  6. 2026Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling
  7. 2026Diffusion Language Models: An Experimental Analysis
  8. 2026Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding