# The Disparate Impacts of Speculative Decoding

arXiv: https://arxiv.org/abs/2510.02128 · HTML: https://mlsanity.com/paper/2510.02128/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 13 of 13)

- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models](https://mlsanity.com/paper/2406.02301/) (2024)
- [Parameter-efficient Modularised Bias Mitigation via AdapterFusion](https://mlsanity.com/paper/2302.06321/) (2023)
- [Phase transition in large language models and the criticality of natural languages](https://mlsanity.com/paper/2406.05335/) (2024)
- [Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters](https://mlsanity.com/paper/2406.16758/) (2024)
- [Low-rank finetuning for LLMs: A fairness perspective](https://mlsanity.com/paper/2405.18572/) (2024)

## Built on by (most cited first, 2 of 2)

- [Speculative Decoding and the Curse of Multilinguality](https://mlsanity.com/paper/2605.30580/) (2026)
- [Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding](https://mlsanity.com/paper/2609.09338/) (2026)

## Abstract (arXiv)

The practice of speculative decoding, whereby inference is probabilistically supported by a smaller, cheaper, ``drafter'' model, has become a standard technique for systematically reducing the decoding time of large language models. This paper conducts an analysis of speculative decoding through the lens of its potential disparate speed-up rates across tasks. Crucially, the paper shows that speed-up gained from speculative decoding is not uniformly distributed across tasks, consistently diminishing for under-fit, and often underrepresented tasks. To better understand this phenomenon, we derive an analysis to quantify this observed ``unfairness'' and draw attention to the factors that motivate such disparate speed-ups to emerge. Further, guided by these insights, the paper proposes a mitigation strategy designed to reduce speed-up disparities and validates the approach across several model pairs, revealing on average a 12% improvement in our fairness metric.