mlsanityData to 14 Oct 2026

From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment

arXiv 2610.04956 ↗First posted 4 Oct 2026cs.CL cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure alternative to public cloud services. However, these resource-constrained servers struggle to guarantee heterogeneous Service Level Objectives (SLOs) when serving multiple LoRA-adapted services simultaneously. Existing serving frameworks suffer from severe SLO violations due to the computational overhead of LoRA layers and the rigid nature of batch scheduling. To address this, we propose HALO, a scheduling method tailored for LoRA-assisted on-premise LLM deployment. HALO introduces two key innovations: a spatial multiplexing strategy that overlaps Base and LoRA computations by partitioning GPU Streaming Multiprocessors (SMs), and an SLO-aware scheduler that decouples request execution based on "request-level slack." By prioritizing urgent tasks and utilizing idle budget for traffic shaping, HALO significantly mitigates resource contention. Our evaluation demonstrates that HALO minimizes SLO violations while improving throughput compared to state-of-the-art baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 43 references that are in the map. First posted on arXiv 4 Oct 2026.

Builds on

20 of 43 references.

References · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2024Yi: Open Foundation Models by 01.AIcited by 719
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  10. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  11. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  12. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
8 more
  1. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 90
  2. 2023Fairness in Serving Large Language Modelscited by 78
  3. 2024CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inferencecited by 22
  4. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  5. 2025A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Servicescited by 17
  6. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 11
  7. 2025Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Managementcited by 5
  8. 2025Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systemscited by 5

Built on by

Not cited yet.