# LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report

arXiv: https://arxiv.org/abs/2405.00732 · HTML: https://mlsanity.com/paper/2405.00732/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://mlsanity.com/paper/2010.11918/) (2020)
- [To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks](https://mlsanity.com/paper/1903.05987/) (2019)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Fine-tuning Large Language Models for Domain-specific Machine Translation](https://mlsanity.com/paper/2402.15061/) (2024)
- [Ludwig: a type-based declarative deep learning toolbox](https://mlsanity.com/paper/1909.07930/) (2019)
- [PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization](https://mlsanity.com/paper/2402.16141/) (2024)
- [Fine-tuning Is a Surprisingly Effective Domain Adaptation Baseline in Handwriting Recognition](https://mlsanity.com/paper/2302.06308/) (2023)

## Built on by (most cited first, 30 of 47)

- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Text-to-LoRA: Instant Transformer Adaption](https://mlsanity.com/paper/2506.06105/) (2025)
- [Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models](https://mlsanity.com/paper/2409.09510/) (2024)
- [Improved Representation Steering for Language Models](https://mlsanity.com/paper/2505.20809/) (2025)
- [LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments](https://mlsanity.com/paper/2406.02611/) (2024)
- [Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs](https://mlsanity.com/paper/2410.13394/) (2024)
- [S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity](https://mlsanity.com/paper/2412.06289/) (2024)
- [SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks](https://mlsanity.com/paper/2506.10424/) (2025)
- [Are Small Language Models Ready to Compete with Large Language Models for Practical Applications?](https://mlsanity.com/paper/2406.11402/) (2024)
- [ICaRus: Identical Cache Reuse for Efficient Multi Model Inference](https://mlsanity.com/paper/2603.13281/) (2026)
- [The Primacy of Magnitude in Low-Rank Adaptation](https://mlsanity.com/paper/2507.06558/) (2025)
- [Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applications](https://mlsanity.com/paper/2407.18990/) (2024)
- [PLoRA: Efficient Concurrent LoRA Training for Large Language Models](https://mlsanity.com/paper/2508.02932/) (2025)
- [Small Language Models for Application Interactions: A Case Study](https://mlsanity.com/paper/2405.20347/) (2024)
- [SDRT: Enhance Vision-Language Models by Self-Distillation with Diverse Reasoning Traces](https://mlsanity.com/paper/2503.01754/) (2025)
- [Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management](https://mlsanity.com/paper/2505.03756/) (2025)
- [DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models](https://mlsanity.com/paper/2509.09097/) (2025)
- [Arctic-TILT. Business Document Understanding at Sub-Billion Scale](https://mlsanity.com/paper/2408.04632/) (2024)
- [HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling](https://mlsanity.com/paper/2508.15919/) (2025)
- [Causal-Guided Detoxify Backdoor Attack of Open-Weight LoRA Models](https://mlsanity.com/paper/2512.19297/) (2025)
- [Scaling Point-in-Time Language Models: Enabling Economic Evaluation of Embeddings](https://mlsanity.com/paper/2607.11889/) (2026)
- [Aligning Large Language Models from Self-Reference AI Feedback with one General Principle](https://mlsanity.com/paper/2406.11190/) (2024)
- [Scrambled text: training Language Models to correct OCR errors using synthetic data](https://mlsanity.com/paper/2409.19735/) (2024)
- [On-Device LLMs for SMEs: Challenges and Opportunities](https://mlsanity.com/paper/2410.16070/) (2024)
- [A Framework for Cryptographic Verifiability of End-to-End AI Pipelines](https://mlsanity.com/paper/2503.22573/) (2025)
- [Insight-RAG: Enhancing LLMs with Insight-Driven Augmentation](https://mlsanity.com/paper/2504.00187/) (2025)
- [Supporting Dynamic Agentic Workloads: How Data and Agents Interact](https://mlsanity.com/paper/2512.09548/) (2025)
- [Mitigating Forgetting in Low Rank Adaptation](https://mlsanity.com/paper/2512.17720/) (2025)
- [Research on Tibetan Tourism Viewpoints information generation system based on LLM](https://mlsanity.com/paper/2407.13561/) (2024)
- [Leveraging Language Models for Emotion and Behavior Analysis in Education](https://mlsanity.com/paper/2408.06874/) (2024)

## Abstract (arXiv)

Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted methods for Parameter Efficient Fine-Tuning (PEFT) of Large Language Models (LLMs). LoRA reduces the number of trainable parameters and memory usage while achieving comparable performance to full fine-tuning. We aim to assess the viability of training and serving LLMs fine-tuned with LoRA in real-world applications. First, we measure the quality of LLMs fine-tuned with quantized low rank adapters across 10 base models and 31 tasks for a total of 310 models. We find that 4-bit LoRA fine-tuned models outperform base models by 34 points and GPT-4 by 10 points on average. Second, we investigate the most effective base models for fine-tuning and assess the correlative and predictive capacities of task complexity heuristics in forecasting the outcomes of fine-tuning. Finally, we evaluate the latency and concurrency capabilities of LoRAX, an open-source Multi-LoRA inference server that facilitates the deployment of multiple LoRA fine-tuned models on a single GPU using shared base model weights and dynamic adapter loading. LoRAX powers LoRA Land, a web application that hosts 25 LoRA fine-tuned Mistral-7B LLMs on a single NVIDIA A100 GPU with 80GB memory. LoRA Land highlights the quality and cost-effectiveness of employing multiple specialized LLMs over a single, general-purpose LLM.