mlsanityData to 7 Oct 2026

OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models

arXiv 2406.01775 ↗First posted 3 Jun 2024cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

The advent of large language models (LLMs) has revolutionized natural language processing, enabling unprecedented capabilities in understanding and generating human-like text. However, the computational cost and convergence times associated with fine-tuning these models remain significant challenges. Low-Rank Adaptation (LoRA) has emerged as a promising method to mitigate these issues by introducing efficient fine-tuning techniques with a reduced number of trainable parameters. In this paper, we present OLoRA, an enhancement to the LoRA method that leverages orthonormal matrix initialization through QR decomposition. OLoRA significantly accelerates the convergence of LLM training while preserving the efficiency benefits of LoRA, such as the number of trainable parameters and GPU memory footprint. Our empirical evaluations demonstrate that OLoRA not only converges faster but also exhibits improved performance compared to standard LoRA across a variety of language modeling tasks. This advancement opens new avenues for more efficient and accessible fine-tuning of LLMs, potentially enabling broader adoption and innovation in natural language applications.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

25 of 31 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
13 more
  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  3. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  4. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  5. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  6. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  7. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  8. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  9. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  10. 2013Predicting Parameters in Deep Learningcited by 223
  11. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  12. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  13. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 19; 2026: 16.

39 papers.

Sort:

By area:

  1. 2025Tensor Product Attention Is All You Needcited by 27
  2. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  3. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13
  4. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 11
  5. 2025On the Convergence Rate of LoRA Gradient Descentcited by 9
  6. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  7. 2024PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakerscited by 8
  8. 2025GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuningcited by 8
  9. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  10. 2024SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Modelscited by 5
  11. 2025CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initializationcited by 5
  12. 2025MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuningcited by 4
27 more
  1. 2025Omni-Mol: Multitask Molecular Model for Any-to-any Modalitiescited by 3
  2. 2025SAS: Simulated Attention Scorecited by 3
  3. 2026On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameterscited by 3
  4. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 2
  5. 2025ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraintscited by 2
  6. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  7. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  8. 2025HRP: High-Rank Preheating for Superior LoRA Initializationcited by 1
  9. 2025LangVision-LoRA-NAS: Neural Architecture Search for Variable LoRA Rank in Vision Language Modelscited by 1
  10. 2025QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Modelscited by 1
  11. 2025Latent Space Factorization in LoRAcited by 1
  12. 2026Multi-Step Semantic Reasoning in Generative Retrievalcited by 1
  13. 2026Unsupervised Adaptation of PDE Foundation Modelscited by 1
  14. 2024OP-LoRA: The Blessing of Dimensionalitycited by 0
  15. 2025A Foundational Individual Mobility Prediction Model based on Open-Source Large Language Modelscited by 0
  16. 2025IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptationcited by 0
  17. 2025Mixtures of SubExperts for Large Language Continual Learningcited by 0
  18. 2026GiVA: Gradient-Informed Bases for Vector-Based Adaptationcited by 0
  19. 2026CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Modelscited by 0
  20. 2026Not How Many, But Which: Parameter Placement in Low-Rank Adaptationcited by 0
  21. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptationcited by 0
  22. 2026GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretrainingcited by 0
  23. 2026SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptationcited by 0
  24. 2026Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVRcited by 0
  25. 2026Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuningcited by 0
  26. 2026TaRA: Training-Aware Low-Rank Adaptation Initializationcited by 0
  27. 2026FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantizationcited by 0