mlsanityData to 7 Oct 2026

Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management

arXiv 2505.03756 ↗First posted 19 Apr 2025cs.AR cs.AI cs.LGcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA adapters in high bandwidth memory of accelerations can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KVs. We therefore propose FASTLIBRA, a Multi-LoRA caching system to optimize the serving performance. FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during the inference with a unified caching pool. The cache swapper determines the swap-in or out of LoRAs and KV caches based on a unified cost model, when the HBM is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 63.4% on average, compared to state-of-the-art works.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 49 references that are in the map. First posted on arXiv 19 Apr 2025.

Builds on

33 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  11. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
19 more
  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  2. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  3. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  4. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  5. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  6. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  7. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  8. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  9. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  10. 2019Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networkscited by 103
  11. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  12. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  13. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 65
  14. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  15. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  16. 2024CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inferencecited by 21
  17. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
  18. 2023The Costly Dilemma: Generalization, Evaluation and Cost-Optimal Deployment of Large Language Modelscited by 5
  19. 2024The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelinescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2
  2. 2026ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cachecited by 2
  3. 2026MinT: Managed Infrastructure for Training and Serving Millions of LLMscited by 1
  4. 2026PreFT: Prefill-only finetuning for efficient inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA
  2. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  3. 2026InfiniLoRA: Disaggregated Multi-LoRA Serving for Large Language Models ↗
  4. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapters
  5. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  7. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  8. 2026LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference