# tLoRA: Efficient Multi-LoRA Training with Elastic Shared Super-Models

arXiv: https://arxiv.org/abs/2602.07263 · HTML: https://mlsanity.com/paper/2602.07263/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 9 of 9)

- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [S-LoRA: Serving Thousands of Concurrent LoRA Adapters](https://mlsanity.com/paper/2311.03285/) (2023)
- [NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://mlsanity.com/paper/2408.12757/) (2024)
- [SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning](https://mlsanity.com/paper/2501.13198/) (2025)
- [Characterization of Large Language Model Development in the Datacenter](https://mlsanity.com/paper/2403.07648/) (2024)
- [Stylus: Automatic Adapter Selection for Diffusion Models](https://mlsanity.com/paper/2404.18928/) (2024)
- [Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters](https://mlsanity.com/paper/2504.17096/) (2025)
- [mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs](https://mlsanity.com/paper/2312.02515/) (2023)
- [Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling](https://mlsanity.com/paper/2409.03365/) (2024)

## Built on by (most cited first, 2 of 2)

- [ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads](https://mlsanity.com/paper/2604.05426/) (2026)
- [MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service](https://mlsanity.com/paper/2605.08527/) (2026)

## Abstract (arXiv)

As Low-Rank Adaptation (LoRA) becomes the standard approach for efficiently fine-tuning large language models (LLMs), shared clusters increasingly execute many concurrent LoRA training jobs over the same frozen backbone. While recent advances enable batching (co-locating) multiple adapters during serving, efficient training-time co-location of heterogeneous LoRA adapters presents unique challenges. Jobs often differ in adapter rank, batch size, and resource allocation, and na"ive batching can introduce synchronization stalls, communication overheads, and per-job slowdowns that are worse than executing independently. We introduce tLoRA, a framework that enables efficient batch training of multiple LoRA jobs. tLoRA fuses adapters that share the same base model into an elastic shared super-model, exploiting existing distributed training frameworks to derive parallelism plans that share resources effectively. At the kernel level, tLoRA employs a fused LoRA kernel that adaptively reconstructs low-rank computation tiles and schedules rank-aware nano-batches to maximize overlap between computation and communication across adapters. At the scheduling layer, tLoRA incorporates an online, residual-capacity-aware scheduler that adaptively groups jobs to maximize collective throughput. Evaluations using real-world cluster traces demonstrate that tLoRA improves training throughput by 1.2--1.8x, job training completion time by 2.3--5.4x, and GPU utilization by 37%.