mlsanityData to 14 Oct 2026

Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions

arXiv 2604.07931 ↗First posted 9 Apr 2026cs.LGcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Output-length prediction is important for efficient LLM serving, as it directly affects batching, memory reservation, and scheduling. For prompt-only length prediction, most existing methods use a one-shot sampled length as the label, implicitly treating each prompt as if it had one true target length. We show that this is unreliable: even under a fixed model and decoding setup, the same prompt induces a prompt-conditioned output length distribution, not a deterministic scalar, and this distribution is consistent with heavy-tailed behavior. Motivated by this, we cast length prediction as robust estimation from heavy-tailed prompt-conditioned length distributions. We propose prompt-conditioned length distribution (ProD) methods, which construct training targets from multiple independent generations of the same prompt. Two variants are developed to reuse the served LLM's hidden states: mboxProD-M, which uses a median-based target for robust point prediction, and ProD-D, which uses a distributional target that preserves prompt-conditioned uncertainty. We provide theoretical justifications by analyzing the estimation error under a surrogate model. Experiments across diverse scenarios show consistent gains in prediction quality.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 25 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

21 of 25 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Predicting LLM Output Length via Entropy-Guided Representations12 Feb
  2. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor14 May
  3. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  4. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  5. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  11. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  12. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
4 more
  1. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 35
  2. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20
  3. 2023Efficient Algorithms for Generalized Linear Bandits with Heavy-tailed Rewardscited by 11
  4. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026OUTLETS: Output-Length Prediction from Speculative Decoding Backbonescited by 1
  2. 2026TokenCast: Forecasting Token Consumption During LLM Agent Executioncited by 0
  3. 2026Denoising Surface: Modeling and Predicting Inference Cost for Diffusion LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inference
  2. 2025ProCut: LLM Prompt Compression via Attribution Estimation
  3. 2026Jumping the Line: Exploiting Length Predictions in LLM Scheduling
  4. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  5. 2026Dynamic Minimax Regret Optimization for Robust LLM Post-Training
  6. 2026Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
  7. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  8. 2026Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling