# CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference

arXiv: https://arxiv.org/abs/2401.11240 · HTML: https://mlsanity.com/paper/2401.11240/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jan 2024.

## Built on by (most cited first, 22 of 22)

- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2404.06448/) (2024)
- [EPIC: Efficient Position-Independent Caching for Serving Large Language Models](https://mlsanity.com/paper/2410.15332/) (2024)
- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models](https://mlsanity.com/paper/2505.02795/) (2025)
- [HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference](https://mlsanity.com/paper/2504.05897/) (2025)
- [Taming the Titans: A Survey of Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.19720/) (2025)
- [EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices](https://mlsanity.com/paper/2507.01438/) (2025)
- [Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments](https://mlsanity.com/paper/2411.17741/) (2024)
- [MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving](https://mlsanity.com/paper/2502.01960/) (2025)
- [Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA](https://mlsanity.com/paper/2512.17910/) (2025)
- [SwiftDiffusion: Efficient Diffusion Model Serving with Add-on Modules](https://mlsanity.com/paper/2407.02031/) (2024)
- [Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices](https://mlsanity.com/paper/2412.20004/) (2024)
- [Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management](https://mlsanity.com/paper/2505.03756/) (2025)
- [MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios](https://mlsanity.com/paper/2603.09983/) (2026)
- [Efficient Serving of LLM Applications with Probabilistic Demand Modeling](https://mlsanity.com/paper/2506.14851/) (2025)
- [A Data-driven ML Approach for Maximizing Performance in LLM-Adapter Serving](https://mlsanity.com/paper/2508.08343/) (2025)
- [Symbiosis: Multi-Adapter Inference and Fine-Tuning](https://mlsanity.com/paper/2507.03220/) (2025)
- [Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving](https://mlsanity.com/paper/2602.24044/) (2026)
- [POLAR: Online Learning for LoRA Adapter Caching and Routing in Edge LLM Serving](https://mlsanity.com/paper/2604.16583/) (2026)
- [Co-occurrence Patterns of LoRA Adapters in Production Diffusion Model Inference Services](https://mlsanity.com/paper/2609.23321/) (2026)
- [From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment](https://mlsanity.com/paper/2610.04956/) (2026)

## Abstract (arXiv)

Pre-trained large language models (LLMs) often need specialization for domain-specific tasks. Low-Rank Adaptation (LoRA) is a popular approach that adapts a base model to multiple tasks by adding lightweight trainable adapters. In this paper, we present CaraServe, a system that efficiently serves many LoRA adapters derived from a common base model. CaraServe maintains the base model on GPUs and dynamically loads activated LoRA adapters from main memory. As GPU loading results in a cold-start that substantially delays token generation, CaraServe employs a CPU-assisted approach. It early starts the activated adapters on CPUs for prefilling as they are being loaded onto GPUs; after loading completes, it then switches to the GPUs for generative LoRA inference. CaraServe develops a highly optimized synchronization mechanism to efficiently coordinate LoRA computation on the CPU and GPU. Moreover, CaraServe employs a rank-aware scheduling algorithm to optimally schedule heterogeneous LoRA requests for maximum service-level objective (SLO) attainment. We have implemented CaraServe and evaluated it against state-of-the-art LoRA serving systems. Our results demonstrate that CaraServe can speed up the average request serving latency by up to 1.4× and achieve an SLO attainment of up to 99%.