Data to 7 Oct 2026

Punica: Multi-Tenant LoRA Serving

arXiv 2310.18547 ↗First posted 28 Oct 2023cs.DC cs.LGcited by 80

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) has become an important and popular method to adapt pre-trained models to specific domains. We present Punica, a system to serve multiple LoRA models in a shared GPU cluster. Punica contains a new CUDA kernel design that allows batching of GPU operations for different LoRA models. This allows a GPU to hold only a single copy of the underlying pre-trained model when serving multiple, different LoRA models, significantly enhancing GPU efficiency in terms of both memory and computation. Our scheduler consolidates multi-tenant LoRA serving workloads in a shared GPU cluster. With a fixed-sized GPU cluster, our evaluations show that Punica achieves 12x higher throughput in serving multiple LoRA models compared to state-of-the-art LLM serving systems while only adding 2ms latency per token. Punica is open source at https://github.com/punica-ai/punica .

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 28 Oct 2023.

Builds on

21 of 37 references.

References · 21

2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621

9 more

2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 24; 2025: 20; 2026: 33.

80 papers.

Sort:

By area:

2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
2024A Survey on LoRA of Large Language Modelscited by 67
2024Post-Training Sparse Attention with Double Sparsitycited by 45
2024NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internalscited by 43
2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
2023DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMscited by 17
2024Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Executioncited by 16

38 more

2024AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Executioncited by 14
2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
2026Learning to Reason in 13 Parameterscited by 13
2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
2024Quantized Delta Weight Is Safety Keepercited by 12
2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
2024PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdificationcited by 8
2024MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shardscited by 7
2026Learning Next Action Predictors from Human-Computer Interactioncited by 7
2025LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bitscited by 6
2024SwiftDiffusion: Efficient Diffusion Model Serving with Add-on Modulescited by 5
2025PiCa: Parameter-Efficient Fine-Tuning with Column Space Projectioncited by 5
2026Metis: Memory Foundation Modelcited by 5
2024TorchOpera: A Compound AI System for LLM Safetycited by 4
2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
2025Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Managementcited by 4
2025Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systemscited by 4
2026OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelismcited by 4
2026Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patchingcited by 4
2025LoRAFusion: Efficient LoRA Fine-Tuning for LLMscited by 3
2025Causal-Guided Detoxify Backdoor Attack of Open-Weight LoRA Modelscited by 3
2026MoLoRA: Composable Specialization via Per-Token Adapter Routingcited by 3
2026SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decodingcited by 3
2026LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budgetcited by 3
2024Computing in the Era of Large Generative Models: From Cloud-Native to AI-Nativecited by 2
2024Institutional Platform for Secure Self-Service Large Language Model Explorationcited by 2
2024ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Modelscited by 2
2024AdaptGCD: Multi-Expert Adapter Tuning for Generalized Category Discoverycited by 2
2024Empower Vision Applications with LoRA LMMcited by 2
2025Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Modelscited by 2
2025Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMscited by 2
2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2
2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challengescited by 2
2026User as Engram: Internalizing Per-User Memory as Local Parametric Editscited by 2
2026PoLoRA: A Preconditioned Orthogonalized LoRA Optimizercited by 2
2025ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Servingcited by 1

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way