mlsanityData to 7 Oct 2026

The Disparate Impacts of Speculative Decoding

arXiv 2510.02128 ↗First posted 2 Oct 2025cs.CL cs.AIcited by 2

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The practice of speculative decoding, whereby inference is probabilistically supported by a smaller, cheaper, ``drafter'' model, has become a standard technique for systematically reducing the decoding time of large language models. This paper conducts an analysis of speculative decoding through the lens of its potential disparate speed-up rates across tasks. Crucially, the paper shows that speed-up gained from speculative decoding is not uniformly distributed across tasks, consistently diminishing for under-fit, and often underrepresented tasks. To better understand this phenomenon, we derive an analysis to quantify this observed ``unfairness'' and draw attention to the factors that motivate such disparate speed-ups to emerge. Further, guided by these insights, the paper proposes a mitigation strategy designed to reduce speed-up disparities and validates the approach across several model pairs, revealing on average a 12% improvement in our fairness metric.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 13 of its 23 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

13 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters24 Jun
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 11

  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  3. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  4. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  5. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  6. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  7. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
  8. 2024mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Modelscited by 31
  9. 2023Parameter-efficient Modularised Bias Mitigation via AdapterFusioncited by 25
  10. 2024Phase transition in large language models and the criticality of natural languagescited by 12
  11. 2024Low-rank finetuning for LLMs: A fairness perspectivecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Speculative Decoding and the Curse of Multilingualitycited by 0
  2. 2026Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  2. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  3. 2024Decoding Speculative Decoding
  4. 2024Faster Cascades via Speculative Decoding
  5. 2025Speculative Decoding: Performance or Illusion?
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  7. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  8. 2026Speculative Decoding with a Speculative Vocabulary