mlsanityData to 7 Oct 2026

Predicting LLM Output Length via Entropy-Guided Representations

arXiv 2602.11812 ↗First posted 12 Feb 2026cs.AIcited by 9

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The long-tailed distribution of sequence lengths in LLM serving and reinforcement learning (RL) sampling causes significant computational waste due to excessive padding in batched inference. Existing methods rely on auxiliary models for static length prediction, but they incur high overhead, generalize poorly, and fail in stochastic "one-to-many" sampling scenarios. We introduce a lightweight framework that reuses the main model's internal hidden states for efficient length prediction. Our framework features two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activations and token entropy for highly accurate static prediction with negligible cost, and 2) Progressive Length Prediction (PLP), which dynamically estimates the remaining length at each decoding step to handle stochastic generation. To validate our approach, we build and release ForeLen, a comprehensive benchmark with long-sequence, Chain-of-Thought, and RL data. On ForeLen, EGTP achieves state-of-the-art accuracy, reducing MAE by 29.16% over the best baseline. Integrating our methods with a length-aware scheduler yields significant end-to-end throughput gains. Our work provides a new technical and evaluation baseline for efficient LLM inference.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 53 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

47 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS8 Sep
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving28 Apr
  3. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  4. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  5. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2017Axiomatic Attribution for Deep Networkscited by 3,350
30 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  7. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  8. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  9. 2024Token-Budget-Aware LLM Reasoningcited by 242
  10. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  11. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  13. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  14. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  15. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  16. 2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
  17. 2024Locate-then-edit for Multi-hop Factual Recall under Knowledge Editingcited by 36
  18. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34
  19. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 33
  20. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 30
  21. 2025Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMscited by 22
  22. 2025Is Your LLM-Based Multi-Agent a Reliable Real-World Planner? Exploring Fraud Detection in Travel Planningcited by 14
  23. 2025Towards User-level Private Reinforcement Learning with Human Feedbackcited by 13
  24. 2025CODEMENV: Benchmarking Large Language Models on Code Migrationcited by 13
  25. 2024MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Languagecited by 9
  26. 2025Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Modelscited by 9
  27. 2025COMPKE: Complex Question Answering under Knowledge Editingcited by 6
  28. 2025Survey of GenAI for Automotive Software Development: From Requirements to Executable Codecited by 5
  29. 2025PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learningcited by 2
  30. 2025Zen-Attention: A Compiler Framework for Dynamic Attention Folding on AMD NPUscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributionscited by 3
  2. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  3. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  4. 2026Length Value Model: Scalable Value Pretraining for Token-Level Length Modelingcited by 0
  5. 2026Clairvoyant: Predictive Shortest-Job-First Admission for Serial LLM Inferencecited by 0
  6. 2026Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practicecited by 0
  7. 2026OUTLETS: Output-Length Prediction from Speculative Decoding Backbonescited by 0
  8. 2026TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Cachingcited by 0
  9. 2026TokenCast: Forecasting Token Consumption During LLM Agent Executioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction
  2. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inference
  3. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning
  4. 2026Compute Where it Counts: Self Optimizing Language Models
  5. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward
  6. 2026Depth-Entropy Guided Sampling for Training-Free LLM Reasoning
  7. 2024On Provable Length and Compositional Generalization
  8. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch