mlsanityData to 7 Oct 2026

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

arXiv 2311.03285 ↗First posted 6 Nov 2023cs.LG cs.AI cs.DCcited by 146

Not on the map. Why?

Cited by 21 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The "pretrain-then-finetune" paradigm is commonly adopted in the deployment of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks, resulting in a substantial collection of LoRA adapters derived from one base model. We observe that this paradigm presents significant opportunities for batched inference during serving. To capitalize on these opportunities, we present S-LoRA, a system designed for the scalable serving of many LoRA adapters. S-LoRA stores all adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. To efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes Unified Paging. Unified Paging uses a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation. Collectively, these features enable S-LoRA to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead. Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with naive support of LoRA serving), S-LoRA can improve the throughput by up to 4 times and increase the number of served adapters by several orders of magnitude. As a result, S-LoRA enables scalable serving of many task-specific fine-tuned models and offers the potential for large-scale customized fine-tuning services. The code is available at https://github.com/S-LoRA/S-LoRA

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Builds on 37 of its 61 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

37 of 61 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
25 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2021GPT Understands, Toocited by 715
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  14. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  15. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  16. 2022Efficiently Scaling Transformer Inferencecited by 529
  17. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  18. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  19. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  20. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  21. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  22. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  23. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  24. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  25. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 54; 2025: 41; 2026: 45.

146 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  3. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  4. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  8. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  9. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 80
  10. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  11. 2023Fairness in Serving Large Language Modelscited by 77
  12. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
38 more
  1. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 60
  4. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  5. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  6. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  7. 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 38
  8. 2024Queue management for slo-oriented large language model servingcited by 36
  9. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  10. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  11. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  12. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 34
  13. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  14. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 30
  15. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  16. 2024LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystemcited by 22
  17. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 22
  18. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  19. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  20. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  21. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 17
  22. 2024LLoCO: Learning Long Contexts Offlinecited by 16
  23. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  24. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  25. 2025A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Servicescited by 15
  26. 2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Modelscited by 14
  27. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  28. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  29. 2024AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Executioncited by 14
  30. 2024PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningcited by 13
  31. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  32. 2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
  33. 2024Quantized Delta Weight Is Safety Keepercited by 12
  34. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  35. 2025RARE: Retrieval-Augmented Reasoning Modelingcited by 11
  36. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  37. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  38. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 10

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024JORA: JAX Tensor-Parallel LoRA Library for Retrieval Augmented Fine-Tuning
  2. 2025Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
  3. 2026ChunkWise LoRA: Adaptive Sequence Partitioning for Memory-Efficient Low-Rank Adaptation and Accelerated LLM Inference
  4. 2026InfiniLoRA: Disaggregated Multi-LoRA Serving for Large Language Models ↗
  5. 2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design
  6. 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA
  7. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  8. 2023Punica: Multi-Tenant LoRA Serving