mlsanityData to 7 Oct 2026

HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling

arXiv 2508.15919 ↗First posted 21 Aug 2025cs.DC cs.AIcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model (LLM) serving faces the dual challenge of meeting strict user-specific service-level objectives (SLOs) while minimizing computational cost under dynamic, multi-task workloads. Existing approaches either rely on static scheduling policies or focus on single-task settings, limiting their applicability in real-world deployments with heterogeneous requests, variable prompt lengths, and elastic scaling requirements. We present HFX, a production LLM serving system that jointly optimizes request scheduling and elastic scaling across model replicas to satisfy diverse SLOs. HFX introduces a scheduler that performs proactive budget estimation and prioritization to ensure SLO compliance for both new and in-flight requests. HFX also integrates a scaler that supports fast device-to-device (D2D) weight transfer, reducing cold-start latency. Additionally, the system supports both colocated and disaggregated prefill/decode deployments, enabling adaptation to diverse workload patterns and cloud environments. Through extensive experiments on multi-task workloads, we demonstrate consistently higher SLO attainment, lower end-to-end latency, and lower NPU usage cost by up to 4.44×, 65.82%, and 49.81%, respectively, compared to state-of-the-art systems. Our results highlight the effectiveness of SLO-aware scheduling and scaling in practical LLM serving, providing a robust framework for cost-efficient and SLO-compliant deployments.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 40 references that are in the map. First posted on arXiv 21 Aug 2025.

Builds on

25 of 40 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving4 Aug
  2. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference29 May
  3. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture17 May
  4. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving10 Apr
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  6. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  7. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving11 May
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  12. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
4 more
  1. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  2. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  3. 2025SLO-Aware Scheduling for Large Language Model Inferencescited by 7
  4. 2025HAS-GPU: Efficient Hybrid Auto-scaling with Fine-grained GPU Allocation for SLO-aware Serverless Inferencescited by 4

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 9
  2. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
  2. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  3. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  4. 2024Queue management for slo-oriented large language model serving
  5. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMs
  6. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling
  7. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
  8. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels