mlsanityData to 7 Oct 2026

Spexis: Speculative Lookahead Scheduling for LLM Inference

arXiv 2609.34370 ↗First posted 28 Sep 2026cs.LG cs.DCcited by 0

New in Efficient LLM inference and serving; no direction yet. Efficient LLM inference and serving

Abstract (arXiv)

Spexis is a multi-GPU LLM inference framework that improves the efficiency of pipeline and tensor parallelism through speculative parallelism. Rather than using speculative decoding only to accelerate token generation, Spexis runs speculation in parallel with normal execution, introducing a new parallelism axis without increasing KV-cache memory usage. This improves memory efficiency and helps mitigate the bottlenecks of multi-GPU inference. Spexis further uses lookahead scheduling to predict speculation quality and future memory pressure, allowing it to reduce wasted speculation, KV-cache eviction, and recomputation. Built on top of vLLM, Spexis largely improves serving performance across a range of GPU configurations, achieving speedups of up to 34% over a baseline that uses the optimal combination of pipeline and tensor parallelism. Spexis's source code is publicly available at https://github.com/mlsys-seo/spexis.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 18 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

12 of 18 references.

References · 12

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  7. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  8. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  9. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  10. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  11. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  12. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  2. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  3. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  4. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
  5. 2026RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation
  6. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  7. 2026SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
  8. 2025Cronus: Efficient LLM inference on Heterogeneous GPU Clusters via Partially Disaggregated Prefill