mlsanityData to 7 Oct 2026

OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs

arXiv 2510.07535 ↗First posted 8 Oct 2025cs.CL cs.AIcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding promises faster inference for large language models (LLMs), yet existing methods fail to generalize to real-world settings. Benchmarks typically assume short contexts (e.g., 2K tokens), whereas practical workloads involve long contexts. We find current approaches degrade severely with long contexts; for instance, EAGLE3 even slows down the generation speed by 0.81x. We address these limitations by releasing a new long-context benchmark (LongSpecBench) and introducing a novel model (OWL). OWL achieves about 5x higher acceptance length than EAGLE3 on long-context inputs through three innovations: (1) an LSTM-based drafter conditioned only on the last-token state, making it generalize to various lengths, (2) a special token [SPEC] in the verifier that produces richer representation for drafter, and (3) a hybrid algorithm combining both tree and non-tree decoding methods. We release all code and datasets to advance future research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 25 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

16 of 25 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications7 Nov
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  4. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput20 Jun
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 8

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  4. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  5. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  6. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  7. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  8. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  2. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  3. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  4. 2025Speculative Decoding: Performance or Illusion?
  5. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
  6. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
  7. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  8. 2025SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences