mlsanityData to 14 Oct 2026

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

arXiv 2508.02932 ↗First posted 4 Aug 2025cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance. While numerous studies have investigated ways to improve LoRA serving efficiency by serving multiple LoRAs concurrently, existing methods assume that a wide range of LoRA adapters are available for serving. In our work, we conduct extensive empirical studies to show that current LoRA training paradigms do not efficiently utilize hardware resources and incur high overhead to obtain a performant LoRA adapter. Leveraging these insights, we propose PLoRA, which automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints and develops performant kernels to improve training efficiency. Across a range of LLMs and LoRA configurations, PLoRA improves training throughput by up to 12.8x and reduces the overall fine-tuning makespan by up to 7.52x compared to existing approaches.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 17 of its 43 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

17 of 43 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  9. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  10. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  11. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  12. 2020Sparse GPU Kernels for Deep Learningcited by 141
5 more
  1. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  2. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  3. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  4. 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
  5. 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  2. 2026A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Searchcited by 2
  3. 2025NanoFlux: Adversarial Dual-LLM Evaluation and Distillation For Multi-Domain Reasoningcited by 1
  4. 2026ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloadscited by 1
  5. 2026Enhancing Language Models for Robust Greenwashing Detectioncited by 0
  6. 2026Leveraging LLMs for Multi-File DSL Code Generation: An Industrial Case Studycited by 0