mlsanityData to 14 Oct 2026

Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving

arXiv 2504.03651 ↗First posted 1 Mar 2025cs.DC cs.AI cs.LGcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models have been widely deployed in various applications, encompassing both interactive online tasks and batched offline tasks. Given the burstiness and latency sensitivity of online tasks, over-provisioning resources is common practice. This allows for the integration of latency-insensitive offline tasks during periods of low online load, enhancing resource utilization. However, strategically serving online and offline tasks through a preemption mechanism fails to fully leverage the flexibility of offline tasks and suffers from KV cache recomputation and irregular workloads. In this paper, we introduce Echo, a collaborative online-offline task serving system, including a scheduler, a KV cache manager, and estimation toolkits. The scheduler and KV cache manager work tightly to maximize the throughput of offline tasks, while the estimator further predicts execution time to ensure online task SLOs. The scheduler leverages the batch information of last iteration to reduce the search space for finding the optimal schedule. The KV cache manager sets the priority of the KV cache based on the type of tasks and the opportunity of prefix sharing to reduce the recomputation. Finally, the estimation toolkits predict the execution time, future memory consumption, and the throughput of offline tasks to guide the scheduler, KV cache manager, and the system deployer. Evaluation based on real-world workloads demonstrates that Echo can increase offline task throughput by up to 3.3×, while satisfying online task SLOs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 62 references that are in the map. First posted on arXiv 1 Mar 2025.

Builds on

29 of 62 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies12 Nov
  2. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  3. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  4. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines18 Mar
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 22

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  6. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2022Efficiently Scaling Transformer Inferencecited by 539
  10. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  11. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  12. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
10 more
  1. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  2. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  3. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  4. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  5. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  6. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  7. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  8. 2024Pie: Pooling CPU Memory for LLM Inferencecited by 13
  9. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  10. 2018Fast and exact analysis for LRU cachescited by 3

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  2. 2025xLLM Technical Reportcited by 3
  3. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  2. 2025OOCO: Latency-disaggregated Architecture for Online-Offline Co-locate LLM Serving
  3. 2025Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization
  4. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
  5. 2026EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference
  6. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  8. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving