mlsanityData to 14 Oct 2026

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

arXiv 2405.12130 ↗First posted 20 May 2024cs.CL cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters. To achieve it, we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA. We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

34 of 41 references.

References · 34

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2020GLU Variants Improve Transformercited by 1,804
22 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  3. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  4. 2023LIMA: Less Is More for Alignmentcited by 1,100
  5. 2023BloombergGPT: A Large Language Model for Financecited by 939
  6. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  8. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  9. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  10. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  11. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  12. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  13. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
  14. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  15. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 124
  16. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  17. 2020Impact of News on the Commodity Market: Dataset and Resultscited by 80
  18. 2023Evaluating Language Models for Mathematics through Interactionscited by 70
  19. 2023DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuningcited by 64
  20. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 50
  21. 2024PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimizationcited by 19
  22. 2024ResLoRA: Identity Residual Mapping in Low-Rank Adaptioncited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 12; 2026: 3.

31 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  3. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuningcited by 48
  4. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  5. 2024QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptationcited by 23
  6. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Datacited by 22
  7. 2024MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mambacited by 11
  8. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 11
  9. 2024DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Modelcited by 8
  10. 2024LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuningcited by 6
  11. 2025Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptationcited by 6
  12. 2025Endo-FASt3r: Endoscopic Foundation model Adaptation for Structure from motioncited by 6
19 more
  1. 2025RaSA: Rank-Sharing Low-Rank Adaptationcited by 6
  2. 2025ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 5
  3. 2025Variational Low-Rank Adaptation for Personalized Impaired Speech Recognitioncited by 4
  4. 2024CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuningcited by 3
  5. 2024Open Llama2 Model for the Lithuanian Languagecited by 2
  6. 20251LoRA: Summation Compression for Very Low-Rank Adaptationcited by 2
  7. 2025Enhancing Low-Rank Adaptation with Structured Nonlinear Transformationscited by 2
  8. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  9. 2024Rapid Biomedical Research Classification: The Pandemic PACT Advanced Categorisation Enginecited by 1
  10. 2024BoRA: Bayesian Hierarchical Low-Rank Adaption for Multi-Task Large Language Modelscited by 1
  11. 2025CURing Large Models: Compression via CUR Decompositioncited by 1
  12. 2025Surgical AI Copilot: Energy-Based Fourier Gradient Low-Rank Adaptation for Surgical LLM Agent Reasoning and Planningcited by 1
  13. 2025Subsampled Randomized Fourier GaLore for Adapting Foundation Models in Depth-Driven Liver Landmark Segmentationcited by 1
  14. 2024Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balancecited by 0
  15. 2024Pear: Pruning and Sharing Adapters in Visual Parameter-Efficient Fine-Tuningcited by 0
  16. 2024Learning from "Silly" Questions Improves Large Language Models, But Only Slightlycited by 0
  17. 2025Fragments to Facts: Partial-Information Fragment Inference from LLMscited by 0
  18. 2026Not How Many, But Which: Parameter Placement in Low-Rank Adaptationcited by 0
  19. 2026What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  2. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  3. 2025LoRA Is Slower Than You Think
  4. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
  5. 2026Flexi-LoRA with Input-Adaptive Ranks: Efficient Finetuning for Speech and Reasoning Tasks
  6. 2025Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
  7. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
  8. 2026Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation