# When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs

arXiv: https://arxiv.org/abs/2411.01076 · HTML: https://mlsanity.com/paper/2411.01076/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 1 Nov 2024.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [Accelerating LLM Inference with Staged Speculative Decoding](https://mlsanity.com/paper/2308.04623/) (2023)
- [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023)
- [SpecTr: Fast Speculative Decoding via Optimal Transport](https://mlsanity.com/paper/2310.15141/) (2023)
- [Spectre Attacks: Exploiting Speculative Execution](https://mlsanity.com/paper/1801.01203/) (2018)
- [Privacy Side Channels in Machine Learning Systems](https://mlsanity.com/paper/2309.05610/) (2023)
- [Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy](https://mlsanity.com/paper/2312.12728/) (2023)
- [Decoding Speculative Decoding](https://mlsanity.com/paper/2402.01528/) (2024)
- [The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems](https://mlsanity.com/paper/2409.20002/) (2024)
- [What Was Your Prompt? A Remote Keylogging Attack on AI Assistants](https://mlsanity.com/paper/2403.09751/) (2024)
- [Remote Timing Attacks on Efficient Language Model Inference](https://mlsanity.com/paper/2410.17175/) (2024)

## Built on by (most cited first, 7 of 7)

- [Auditing Prompt Caching in Language Model APIs](https://mlsanity.com/paper/2502.07776/) (2025)
- [Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding](https://mlsanity.com/paper/2605.14005/) (2026)
- [Adversarial Prompts for Acceptance Collapse in Speculative Decoding](https://mlsanity.com/paper/2607.21804/) (2026)
- [Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Tools](https://mlsanity.com/paper/2606.02483/) (2026)
- [Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion](https://mlsanity.com/paper/2606.25097/) (2026)
- [Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing](https://mlsanity.com/paper/2607.20723/) (2026)
- [Speedbumps: Rejection Attacks on Speculative Decoding](https://mlsanity.com/paper/2610.10929/) (2026)

## Abstract (arXiv)

Deployed large language models (LLMs) often rely on speculative decoding, a technique that generates and verifies multiple candidate tokens in parallel, to improve throughput and latency. In this work, we reveal a new side-channel whereby input-dependent patterns of correct and incorrect speculations can be inferred by monitoring per-iteration token counts or packet sizes. In evaluations using research prototypes and production-grade vLLM serving frameworks, we show that an adversary monitoring these patterns can fingerprint user queries (from a set of 50 prompts) with over 75% accuracy across four speculative-decoding schemes at temperature 0.3: REST (100%), LADE (91.6%), BiLD (95.2%), and EAGLE (77.6%). Even at temperature 1.0, accuracy remains far above the 2% random baseline - REST (99.6%), LADE (61.2%), BiLD (63.6%), and EAGLE (24%). We also show the capability of the attacker to leak confidential datastore contents used for prediction at rates exceeding 25 tokens/sec. To defend against these, we propose and evaluate a suite of mitigations, including packet padding and iteration-wise token aggregation.