mlsanityData to 14 Oct 2026

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

arXiv 2609.01068 ↗First posted 1 Sep 2026cs.CLcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing approaches have key drawbacks: external proxy models add substantial latency and often have limited fidelity, whereas internal state-based methods are efficient but rely on shallow probes of current model states. We identify a structural connection between speculative decoding (SD) and length prediction: latent representations produced by the draft decoder in advanced frameworks (e.g., EAGLE-3) encode signals that are predictive of generation length. Building on this insight, we introduce OUTLETS (Output-Length Prediction from Speculative Decoding Backbones), which repurposes the speculative backbone as a trajectory-aware length predictor. When its draft representations are already computed for speculative decoding, OUTLETS adds only a lightweight regression head and achieves lower MAE than the evaluated methods. Under saturated disaggregated serving, OUTLETS predictions enable standard scheduling policies to prioritize shorter requests and distribute requests more evenly across decoding instances, reducing short-request P99 latency by 34.8%.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 41 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

27 of 41 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding19 Mar
  2. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios25 Nov
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  5. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  6. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  7. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  10. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  8. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  9. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  10. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  11. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  12. 2019Scheduling with Predictions and the Price of Mispredictioncited by 47
5 more
  1. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 35
  2. 2025Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMscited by 22
  3. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  4. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 8
  5. 2026Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributionscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Jumping the Line: Exploiting Length Predictions in LLM Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  2. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  3. 2025Speculative Decoding: Performance or Illusion?
  4. 2026Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
  5. 2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models
  6. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
  7. 2025OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs
  8. 2023Accelerating LLM Inference with Staged Speculative Decoding