mlsanityData to 7 Oct 2026

Asymmetry in Low-Rank Adapters of Foundation Models

arXiv 2402.16842 ↗First posted 26 Feb 2024cs.LGcited by 81

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient fine-tuning optimizes large, pre-trained foundation models by updating a subset of parameters; in this class, Low-Rank Adaptation (LoRA) is particularly effective. Inspired by an effort to investigate the different roles of LoRA matrices during fine-tuning, this paper characterizes and leverages unexpected asymmetry in the importance of low-rank adapter matrices. Specifically, when updating the parameter matrices of a neural network by adding a product BA, we observe that the B and A matrices have distinct functions: A extracts features from the input, while B uses these features to create the desired output. Based on this observation, we demonstrate that fine-tuning B is inherently more effective than fine-tuning A, and that a random untrained A should perform nearly as well as a fine-tuned one. Using an information-theoretic lens, we also bound the generalization of low-rank adapters, showing that the parameter savings of exclusively training B improves the bound. We support our conclusions with experiments on RoBERTa, BART-Large, LLaMA-2, and ViTs.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

28 of 40 references.

References · 28

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2020In Search of Lost Domain Generalizationcited by 1,042
16 more
  1. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  4. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  5. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  6. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  7. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  8. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  9. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  10. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  11. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  12. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  13. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  14. 2016A Thorough Examination of the CNN/Daily Mail Reading Comprehension Taskcited by 71
  15. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  16. 2024PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 41; 2026: 23.

81 papers.

Sort:

By area:

  1. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  2. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  3. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  4. 2024A Survey on LoRA of Large Language Modelscited by 67
  5. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  6. 2025MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generationcited by 43
  7. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  8. 2025Scaling Laws for Forgetting during Finetuning with Pretraining Data Injectioncited by 35
  9. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  10. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  11. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 26
  12. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25
38 more
  1. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  2. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  3. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  4. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  5. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  6. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 13
  7. 2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 12
  8. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 12
  9. 2024SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modulescited by 10
  10. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  11. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  12. 2025Riemannian Optimization for LoRA on the Stiefel Manifoldcited by 10
  13. 2024Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimizationcited by 9
  14. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  15. 2025DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Modelscited by 9
  16. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 9
  17. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 9
  18. 2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learningcited by 7
  19. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
  20. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  21. 2025HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Mergingcited by 6
  22. 2024Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Modelscited by 5
  23. 2025CoLA: Collaborative Low-Rank Adaptationcited by 5
  24. 2025Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Modelscited by 4
  25. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  26. 2025HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalizationcited by 4
  27. 2025LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysiscited by 4
  28. 2024LoRA-Mini : Adaptation Matrices Decomposition and Selective Trainingcited by 3
  29. 2024SAMCL: Empowering SAM to Continually Learn from Dynamic Domains with Extreme Storage Efficiencycited by 3
  30. 2025Variational Low-Rank Adaptation for Personalized Impaired Speech Recognitioncited by 3
  31. 2025Rate-My-LoRA: Efficient and Adaptive Federated Model Tuning for Cardiac MRI Segmentationcited by 2
  32. 2025DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Modelscited by 2
  33. 2025Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptationcited by 2
  34. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  35. 2026Soro: A Lightweight Foundation Model and Chatbot for Tajikcited by 2
  36. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 2
  37. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  38. 2025Task-agnostic Low-rank Residual Adaptation for Efficient Federated Continual Fine-Tuningcited by 1

50 most cited of 81 All 81 (JSON)