mlsanityData to 7 Oct 2026

TimeBill: Time-Budgeted Inference for Large Language Models

arXiv 2512.21859 ↗First posted 26 Dec 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed in time-critical systems, such as robotics, autonomous driving, embodied intelligence, and industrial automation, where generating accurate responses within a given time budget is crucial for decision-making, control, or safety-critical tasks. However, the auto-regressive generation process of LLMs makes it challenging to model and estimate the end-to-end execution time. Furthermore, existing efficient inference methods based on a fixed key-value (KV) cache eviction ratio struggle to adapt to varying tasks with diverse time budgets, where an improper eviction ratio may lead to incomplete inference or a drop in response performance. In this paper, we propose TimeBill, a novel time-budgeted inference framework for LLMs that balances the inference efficiency and response performance. To be more specific, we propose a fine-grained response length predictor (RLP) and an execution time estimator (ETE) to accurately predict the end-to-end execution time of LLMs. Following this, we develop a time-budgeted efficient inference approach that adaptively adjusts the KV cache eviction ratio based on execution time prediction and the given time budget. Finally, through extensive experiments, we demonstrate the advantages of TimeBill in improving task completion rate and maintaining response performance under various overrun strategies.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 26 Dec 2025.

Builds on

20 of 35 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2019Root Mean Square Layer Normalizationcited by 1,425
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
8 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  5. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  6. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  7. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  8. 2025BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architecturescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Timely Machine: Awareness of Time Makes Test-Time Scaling Agenticcited by 5
  2. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  3. 2026Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMscited by 0
  4. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Online Scheduling for LLM Inference with KV Cache Constraints
  2. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  3. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  6. 2024A Survey on Efficient Inference for Large Language Models
  7. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
  8. 2025Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity