mlsanityData to 7 Oct 2026

HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location

arXiv 2501.14808 ↗First posted 15 Jan 2025cs.DC cs.LGcited by 10

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) have facilitated a wide range of applications with distinct service-level objectives (SLOs), from latency-sensitive online tasks like interactive chatbots to throughput-oriented offline workloads like data synthesis. The existing deployment model, which dedicates machines to each workload, simplifies SLO management but often leads to poor resource utilization. This paper introduces HyGen, an interference-aware LLM serving system that enables efficient co-location of online and offline workloads while preserving SLOs. HyGen incorporates two key innovations: (1) performance control mechanisms, including a latency predictor to estimate batch execution time and an SLO-aware profiler to quantify latency interference, and (2) SLO-aware offline scheduling policies that maximize serving throughput and prevent starvation. Our evaluation on production workloads shows that HyGen achieves up to 3.9-5.8x throughput gains over online and hybrid serving baselines, while ensuring latency SLOs. The code of HyGen is publicly available at https://github.com/UIUC-MLSys/HyGen.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 55 of its 97 references that are in the map. First posted on arXiv 15 Jan 2025.

Builds on

55 of 97 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  2. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  3. 2024Queue management for slo-oriented large language model serving5 Jun
  4. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  5. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  6. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving2 Apr
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  8. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inference2 Feb
  9. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
  10. 2023Fairness in Serving Large Language Models31 Dec

Other references · 45

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
33 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2015Teaching Machines to Read and Comprehendcited by 1,039
  3. 2020Aligning AI With Shared Human Valuescited by 706
  4. 2024Yi: Open Foundation Models by 01.AIcited by 706
  5. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2022Efficiently Scaling Transformer Inferencecited by 529
  9. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  13. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  14. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  15. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  16. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  17. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  18. 2022Can Foundation Models Wrangle Your Data?cited by 173
  19. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  20. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  21. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  22. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  23. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  24. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  25. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  26. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  27. 2023SummIt: Iterative Text Summarization via ChatGPTcited by 53
  28. 2021SpreadsheetCoder: Formula Prediction from Semi-structured Contextcited by 41
  29. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  30. 2024Mirage: A Multi-Level Superoptimizer for Tensor Programscited by 39
  31. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
  32. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  33. 2021Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problemcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  2. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraintscited by 3
  3. 2025xLLM Technical Reportcited by 3
  4. 2025DiSCo: Device-Server Collaborative LLM-Based Text Streaming Servicescited by 2
  5. 2025OOCO: Latency-disaggregated Architecture for Online-Offline Co-locate LLM Servingcited by 1
  6. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  7. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  8. 2026AutoScout: Structured Optimization for Automating ML System Configurationcited by 0
  9. 2026Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNICcited by 0
  10. 2026Inference-Native Zeroth-Order Optimization for LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling
  2. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information
  3. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  4. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
  5. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production
  6. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  7. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
  8. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving