mlsanityData to 7 Oct 2026

When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs

arXiv 2411.01076 ↗First posted 1 Nov 2024cs.CL cs.AI cs.CRcited by 6

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Deployed large language models (LLMs) often rely on speculative decoding, a technique that generates and verifies multiple candidate tokens in parallel, to improve throughput and latency. In this work, we reveal a new side-channel whereby input-dependent patterns of correct and incorrect speculations can be inferred by monitoring per-iteration token counts or packet sizes. In evaluations using research prototypes and production-grade vLLM serving frameworks, we show that an adversary monitoring these patterns can fingerprint user queries (from a set of 50 prompts) with over 75% accuracy across four speculative-decoding schemes at temperature 0.3: REST (100%), LADE (91.6%), BiLD (95.2%), and EAGLE (77.6%). Even at temperature 1.0, accuracy remains far above the 2% random baseline - REST (99.6%), LADE (61.2%), BiLD (63.6%), and EAGLE (24%). We also show the capability of the attacker to leak confidential datastore contents used for prediction at rates exceeding 25 tokens/sec. To defend against these, we propose and evaluate a suite of mitigations, including packet padding and iteration-wise token aggregation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 39 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

28 of 39 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Remote Timing Attacks on Efficient Language Model Inference22 Oct
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  3. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  4. 2024Decoding Speculative Decoding2 Feb
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  7. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  8. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  9. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  10. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  11. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  12. 2023Speculative Decoding with Big Little Decoder15 Feb
2 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  9. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  10. 2018Spectre Attacks: Exploiting Speculative Executioncited by 47
  11. 2023Privacy Side Channels in Machine Learning Systemscited by 38
  12. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
2 more
  1. 2024The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systemscited by 29
  2. 2024What Was Your Prompt? A Remote Keylogging Attack on AI Assistantscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Auditing Prompt Caching in Language Model APIscited by 19
  2. 2026Adversarial Prompts for Acceptance Collapse in Speculative Decodingcited by 1
  3. 2026Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Toolscited by 0
  4. 2026Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansioncited by 0
  5. 2026Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timingcited by 0
  6. 2026Speedbumps: Rejection Attacks on Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Secure Speculative Decoding for Large Language Models
  2. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  3. 2025Speculative Decoding: Performance or Illusion?
  4. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models
  5. 2026Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
  6. 2025Speculative Safety-Aware Decoding
  7. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
  8. 2026SpecGuard: Inference-Time Backdoor Detection For Free