mlsanityData to 14 Oct 2026

SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference

arXiv 2605.08151 ↗First posted 4 May 2026cs.DC cs.AIcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

LLM serving platforms are increasingly deployed as multi-model cloud systems, where user demand is often long-tailed: a few popular large models receive most requests, while many smaller tail models remain underutilized. We propose SPECTRE (Parallel SPECulative Decoding with a Multi-Tenant REmote Drafter), a serving framework that reuses underutilized tail-model services as remote drafters for heavily loaded large-model services through speculative decoding. SPECTRE enables draft generation and target-side verification to run in parallel, and makes such parallelism effective through three techniques: a hybrid ordinary-parallel speculative decoding strategy guided by a threshold derived from throughput analysis, speculative priority scheduling to preserve draft--target overlap under multi-tenant traffic, and draft-side prompt compression to reduce draft latency. We implement SPECTRE in textttSGLang and evaluate it across multiple draft--target model pairs, reasoning benchmarks, real-world long-context workloads, and a wide range of batch sizes. Results show that SPECTRE consistently improves large-model serving throughput while causing only minor interference to the native workloads of tail-model services. In large-model deployments, including Qwen3-235B-A22B with TP=8, SPECTRE achieves up to 2.28× speedup over autoregressive decoding and up to an additional 66% relative improvement over the strongest speculative decoding baselines. Talk is cheap, we show you the code: https://github.com/sgl-project/sglang/pull/22272.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 37 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

31 of 37 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding16 Apr
  2. 2026Speculative Speculative Decoding3 Mar
  3. 2026MineDraft: A Framework for Batch Parallel Speculative Decoding24 Feb
  4. 2026Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism9 Jan
  5. 2025Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models30 Oct
  6. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  7. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  9. 2024Decoding Speculative Decoding2 Feb
  10. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  11. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
3 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  3. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  12. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
4 more
  1. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  3. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 88
  4. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 52

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  2. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  3. 2025SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding
  4. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  5. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
  6. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models
  7. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
  8. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving