mlsanityData to 7 Oct 2026

Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes

arXiv 2608.03839 ↗First posted 4 Aug 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context. That lookup misses correct drafts already in the pool, most visibly on tool-calling traffic, where a request repeats almost everything but the few values minted for it, and where one rejected token discards the correct continuation behind it. We diagnose the failure position by position across ten benchmarks and find it to be a problem of addressing rather than of coverage: on our densest tool-calling benchmark, about half of what the strongest exact-match drafter misses is present in the pool yet unreachable by exact matching. We therefore propose a second, semantic draft source: the same pool, re-keyed by the hidden state the verifier has already computed at each committed token, together with a merge that lets it ride inside an existing lexical drafter's tree. In three published drafters, at matched pool and budget, it lifts accepted length by 24-29%. Oilbird reaches 4.4x autoregressive decoding speed on API-Bank, against 3.9x for the strongest training-free baseline in our harness and 2.0x for EAGLE-3.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 53 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

43 of 53 references.

Closest ancestors · 25

Same direction, newest first

  1. 2026A Practical Investigation of Training-free Relaxed Speculative Decoding9 Jul
  2. 2026AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding4 Jun
  3. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding19 May
  4. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding16 Apr
  5. 2026ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding15 Apr
  6. 2026SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding14 Apr
  7. 2026Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding2 Apr
  8. 2026Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States3 Feb
  9. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match28 Nov
  10. 2025When, What, and How: Rethinking Retrieval-Enhanced Speculative Decoding3 Nov
  11. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation2 Jul
  12. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
13 more
  1. 2025DReSD: Dense Retrieval for Speculative Decoding21 Feb
  2. 2024SAM Decoding: Speculative Decoding via Suffix Automaton16 Nov
  3. 2024CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decoding8 Aug
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  5. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attribution29 May
  6. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  7. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  9. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  10. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  11. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  12. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  13. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 18

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  8. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  9. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  10. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  11. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  12. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
6 more
  1. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  2. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  3. 2021Efficient Nearest Neighbor Language Modelscited by 93
  4. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  5. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  6. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding
  2. 2025Draft, Verify, and Improve: Toward Training-Aware Speculative Decoding
  3. 2026LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
  4. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
  5. 2026DEdit: Iterative Draft Editing for Speculative Decoding
  6. 2026Carryover Drafting: Recycling Rejected States for Speculative Decoding
  7. 2026Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
  8. 2026DLoop: Looped Speculative Decoding