Data to 7 Oct 2026

The Expressive Power of Low-Rank Adaptation

arXiv 2310.17513 ↗First posted 26 Oct 2023cs.LG cs.AI cs.CLcited by 100

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that leverages low-rank adaptation of weight matrices, has emerged as a prevalent technique for fine-tuning pre-trained models such as large language models and diffusion models. Despite its huge success in practice, the theoretical underpinnings of LoRA have largely remained unexplored. This paper takes the first step to bridge this gap by theoretically analyzing the expressive power of LoRA. We prove that, for fully connected neural networks, LoRA can adapt any model f to accurately represent any smaller target model overlinef if LoRA-rank ≥(width of f) × fracdepth of overlinefdepth of f. We also quantify the approximation error when LoRA-rank is lower than the threshold. For Transformer networks, we show any model can be adapted to a target model of the same size with rank-(fracembedding size2) LoRA adapters.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 59 of its 79 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

59 of 79 references.

References · 59

2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2020Language Models are Few-Shot Learnerscited by 32,199
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234

38 more

2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
2022Transformers learn in-context by gradient descentcited by 714
2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
2018Do Better ImageNet Models Transfer Better?cited by 560
2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
2016On the Expressive Power of Deep Neural Networkscited by 302
2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
2016Benefits of depth in neural networkscited by 248
2016Deep Learning without Poor Local Minimacited by 233
2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
2023Looped Transformers as Programmable Computerscited by 224
2020Few-Shot Learning via Learning the Representation, Provablycited by 213
2015The Benefit of Multitask Representation Learningcited by 194
2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
2015The Power of Depth for Feedforward Neural Networkscited by 181
2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
2023The Learnability of In-Context Learningcited by 126
2014Return of the Devil in the Details: Delving Deep into Convolutional Netscited by 122
2019Deep Network Approximation Characterized by Number of Neuronscited by 120
2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
2018Adversarial Reprogramming of Neural Networkscited by 115
2016Identity Matters in Deep Learningcited by 107
2020Minimum Width for Universal Approximationcited by 92
2015Representation Benefits of Deep Feedforward Networkscited by 84
2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
2016Why Deep Neural Networks for Function Approximation?cited by 82
2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 82
2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 35; 2025: 36; 2026: 27.

100 papers.

Sort:

By area:

2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
2024LoRA Learns Less and Forgets Lesscited by 313
2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Modelcited by 112
2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
2024A Survey on LoRA of Large Language Modelscited by 67
2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
2024How Much Can RAG Help the Reasoning of LLM?cited by 31
2024The Fine-Grained Complexity of Gradient Computation for Training Large Language Modelscited by 30

38 more

2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 29
2025Tensor Product Attention Is All You Needcited by 27
2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 21
2024GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learningcited by 21
2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 19
2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
2024ResLoRA: Identity Residual Mapping in Low-Rank Adaptioncited by 18
2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
2024The effect of fine-tuning on language model toxicitycited by 15
2024textitTrans-LoRA: towards data-free Transferable Parameter Efficient Finetuningcited by 13
2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
2024Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Modelscited by 11
2024Parameter-Efficient Fine-Tuning of State Space Modelscited by 11
2024Towards Infinite-Long Prefix in Transformercited by 10
2024YOLO-UniOW: Efficient Universal Open-World Object Detectioncited by 9
2025Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transformcited by 9
2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 9
2025On the Convergence Rate of LoRA Gradient Descentcited by 9
2025Fast and Low-Cost Genomic Foundation Models via Outlier Removalcited by 8
2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuningcited by 8
2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 6
2025RaSA: Rank-Sharing Low-Rank Adaptationcited by 6
2025Diffusion Model Quantization: A Reviewcited by 6
2024Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapterscited by 5
2024FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clusteringcited by 5
2024Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMscited by 5
2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 5
2024Parameter-Efficient Fine-Tuning via Circular Convolutioncited by 4
2025Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?cited by 4
2025Probabilistic Forecasting for Building Energy Systems using Time-Series Foundation Modelscited by 4
2025BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversitycited by 4
2024LoRA-Mini : Adaptation Matrices Decomposition and Selective Trainingcited by 3
2025FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregationcited by 3
2025SAS: Simulated Attention Scorecited by 3

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way