mlsanityData to 7 Oct 2026

Low-Rank Adaptation for Foundation Models: A Comprehensive Review

arXiv 2501.00365 ↗First posted 31 Dec 2024cs.LG cs.AIcited by 35

Not on the map. Why?

Abstract (arXiv)

The rapid advancement of foundation modelslarge-scale neural networks trained on diverse, extensive datasetshas revolutionized artificial intelligence, enabling unprecedented advancements across domains such as natural language processing, computer vision, and scientific discovery. However, the substantial parameter count of these models, often reaching billions or trillions, poses significant challenges in adapting them to specific downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a highly promising approach for mitigating these challenges, offering a parameter-efficient mechanism to fine-tune foundation models with minimal computational overhead. This survey provides the first comprehensive review of LoRA techniques beyond large Language Models to general foundation models, including recent techniques foundations, emerging frontiers and applications of low-rank adaptation across multiple domains. Finally, this survey discusses key challenges and future research directions in theoretical understanding, scalability, and robustness. This survey serves as a valuable resource for researchers and practitioners working with efficient foundation model adaptation.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 188 of its 244 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

188 of 244 references.

References · 188

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2023Segment Anythingcited by 8,158
38 more
  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  13. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  14. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  15. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  16. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  17. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  18. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  19. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  20. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  21. 2017Mixed Precision Trainingcited by 1,135
  22. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  23. 2021Ethical and social risks of harm from Language Modelscited by 957
  24. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  25. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  26. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  27. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  28. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  29. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  30. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  31. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  32. 2020Time Series Forecasting With Deep Learning: A Surveycited by 457
  33. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  34. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  35. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  36. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  37. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  38. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359

50 most cited of 188 All 188 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 16.

35 papers.

Sort:

By area:

  1. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  2. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
  3. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 29
  4. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  5. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  6. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Taskscited by 10
  7. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  8. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  9. 2025Diffusion Model Quantization: A Reviewcited by 6
  10. 2025ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needscited by 5
  11. 2025Sparse Activation Editing for Reliable Instruction Following in Narrativescited by 3
  12. 2025Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAscited by 3
23 more
  1. 2026"Unlimited Realm of Exploration and Experimentation": Methods and Motivations of AI-Generated Sexual Content Creatorscited by 3
  2. 2025Benchmarking Foundation Models for Mitotic Figure Classificationcited by 2
  3. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  4. 2025Polar Perspectives: Evaluating 2-D LiDAR Projections for Robust Place Recognition with Visual Foundation Modelscited by 2
  5. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  6. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  7. 2026Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Taxcited by 2
  8. 2025LoRMA: Low-Rank Multiplicative Adaptation for LLMscited by 1
  9. 2025LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimizationcited by 1
  10. 2025Hypernetwork-Driven Low-Rank Adaptation Across Attention Headscited by 1
  11. 2026Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimizationcited by 1
  12. 2026Memory Has Geometry: Non-Uniform Geometric Memory for Long-Horizon Personalized AIcited by 1
  13. 2025Bringing Multi-Modal Multi-Task Federated Foundation Models to Education Domain: Prospects and Challengescited by 0
  14. 2025Comparative Analysis of LoRA-Adapted Embedding Models for Clinical Cardiology Text Representationcited by 0
  15. 2026EventFace: Event-Based Face Recognition via Structure-Driven Spatiotemporal Modelingcited by 0
  16. 2026An empirical study of LoRA-based fine-tuning of large language models for automated test case generationcited by 0
  17. 2026Domain Adaptation of Large Language Models for Polymer-Composite Additive Manufacturing Using Retrieval-Augmented Generation and Fine-Tuningcited by 0
  18. 2026Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapterscited by 0
  19. 2026The Hidden Power of Scaling Factor in LoRA Optimizationcited by 0
  20. 2026PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thoughtcited by 0
  21. 2026ROS2SmolVLA: Enabling Small Vision-Language-Action Models for Integration into Industrial-Grade Lightweight Robotscited by 0
  22. 2026Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solutioncited by 0
  23. 2026Generative AI-Based Data Augmentation for Oral Lesion Classification: The PhotoMOCI Dataset and Benchmarkcited by 0