# Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management

arXiv: https://arxiv.org/abs/2505.03756 · HTML: https://mlsanity.com/paper/2505.03756/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 19 Apr 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.10512/) (2023)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://mlsanity.com/paper/1909.05358/) (2019)
- [Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks](https://mlsanity.com/paper/1909.09586/) (2019)
- [Punica: Multi-Tenant LoRA Serving](https://mlsanity.com/paper/2310.18547/) (2023)
- [LoRA-Pro: Are Low-Rank Adapters Properly Optimized?](https://mlsanity.com/paper/2407.18242/) (2024)
- [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://mlsanity.com/paper/2003.03423/) (2020)
- [Stateful Large Language Model Serving with Pensieve](https://mlsanity.com/paper/2312.05516/) (2023)
- [LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report](https://mlsanity.com/paper/2405.00732/) (2024)
- [Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arena](https://mlsanity.com/paper/2407.10627/) (2024)
- [CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference](https://mlsanity.com/paper/2401.11240/) (2024)

## Built on by (most cited first, 5 of 5)

- [ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management](https://mlsanity.com/paper/2601.21473/) (2026)
- [ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache](https://mlsanity.com/paper/2604.06370/) (2026)
- [MinT: Managed Infrastructure for Training and Serving Millions of LLMs](https://mlsanity.com/paper/2605.13779/) (2026)
- [PreFT: Prefill-only finetuning for efficient inference](https://mlsanity.com/paper/2605.14217/) (2026)
- [From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment](https://mlsanity.com/paper/2610.04956/) (2026)

## Abstract (arXiv)

Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA adapters in high bandwidth memory of accelerations can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KVs. We therefore propose FASTLIBRA, a Multi-LoRA caching system to optimize the serving performance. FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during the inference with a unified caching pool. The cache swapper determines the swap-in or out of LoRAs and KV caches based on a unified cost model, when the HBM is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 63.4% on average, compared to state-of-the-art works.