mlsanityData to 7 Oct 2026

LoRA: Low-Rank Adaptation of Large Language Models

arXiv 2106.09685 ↗First posted 17 Jun 2021cs.CL cs.AI cs.LGcited by 14,084

Not on the map. Why?

Cited by 1,272 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

Key facts. Cited by 14,084 papers in the map (data through 7 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

43 of 65 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
31 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  6. 2018Neural Network Acceptability Judgmentscited by 952
  7. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  9. 2021GPT Understands, Toocited by 715
  10. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  11. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  12. 2017Learning multiple visual domains with residual adapterscited by 553
  13. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  15. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  16. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  17. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  18. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  19. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  20. 2013Predicting Parameters in Deep Learningcited by 223
  21. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  22. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  23. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  24. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  25. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  26. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  27. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  28. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  29. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  30. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  31. 2016Recovery guarantee of weighted low-rank approximation via alternating minimizationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 178; 2023: 1605; 2024: 4291; 2025: 4334; 2026: 3657.

14,084 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
38 more
  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  5. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  6. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  9. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  13. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  14. 2021Learning to Prompt for Continual Learningcited by 938
  15. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  16. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  17. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  18. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  19. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  20. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  21. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  22. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  23. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  24. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  25. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  26. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  27. 2023Adversarial Diffusion Distillationcited by 741
  28. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  29. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  30. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  31. 2024Chronos: Learning the Language of Time Seriescited by 712
  32. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  33. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  34. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  35. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  36. 2023ModelScope Text-to-Video Technical Reportcited by 651
  37. 2024A Survey on Large Language Models for Code Generationcited by 646
  38. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643

50 most cited of 14,084 All 14,084 (JSON)