mlsanityData to 14 Oct 2026

LAPS: A Length-Aware-Prefill LLM Serving System

arXiv 2601.11589 ↗First posted 4 Jan 2026cs.DC cs.AIcited by 4

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

LAPS identifies and disaggregates requests with different prompt lengths in LLM serving to reduce TTFT latency. While recent systems have decoupled the prefill and decode stages to improve throughput, they still rely on unified scheduling policies that fail to adapt to heterogeneous workload characteristics. We observe that prompt-length variations lead to distinct performance bottlenecks, motivating an adaptive scheduling strategy. LAPS disaggregates multi-turn long-prefill requests from short-prefill ones and introduces a length-aware smart batching mechanism for short-prefill workloads. It adopts a dual-queue design that supports temporal disaggregation on a single prefill instance or spatial disaggregation across multiple instances. For short-prefill batches, a batch waiting window and CUDA Graph-based clustering mitigate interference from heterogeneous computation, reducing batching delay and lowering average latency. In real multi-turn workloads, LAPS reduces prefill latency by over 30% compared to vanilla SGLang under prefill-decode disaggregation, and further decreases SLO violations by 28% in multi-instance deployments with vanilla data-parallel configuration. Compared to the SGLang router with load balancing, it further lowers SLO violations by 12% in multi-GPU settings. Under high concurrency and mixed-request scenarios, LAPS improves request throughput by 35% serving Qwen2.5-32B model for prefill instance, demonstrating its effectiveness in optimizing heterogeneous LLM serving workloads.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 30 references that are in the map. First posted on arXiv 4 Jan 2026.

Builds on

19 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving23 Jul
  2. 2025Universal Model Routing for Efficient LLM Inference12 Feb
  3. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inference22 Aug
  4. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 13

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  5. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  6. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  7. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  8. 2023A Comprehensive Survey of Dataset Distillationcited by 115
  9. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  10. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  11. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  12. 2024Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancingcited by 7
1 more
  1. 2025Token Level Routing Inference System for Edge Devicescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanismcited by 3
  2. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  3. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Servingcited by 0
  4. 2026Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  3. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
  4. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  5. 2026Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving
  6. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  7. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  8. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inference