mlsanityData to 7 Oct 2026

Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

arXiv 2507.02503 ↗First posted 3 Jul 2025cs.LG cs.AI cs.CEcited by 10

Not on the map. Why?

Abstract (arXiv)

Continual fine-tuning of Large Language Models (LLMs) is hampered by the trade-off between efficiency and expressiveness. Low-Rank Adaptation (LoRA) offers efficiency but constrains the model's ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. We propose GORP (Gradient LOw Rank Projection) for Continual Learning, a novel training strategy that overcomes these limitations by synergistically combining full and low-rank parameters and jointly updating within a unified low-rank gradient subspace. GORP expands the optimization space while preserving efficiency and mitigating catastrophic forgetting. Extensive experiments on continual learning benchmarks demonstrate GORP's superior performance compared to existing state-of-the-art approaches. Code is available at https://github.com/Wcxwcxw/GORP.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 40 of its 47 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

40 of 47 references.

References · 40

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2016Learning without Forgettingcited by 2,465
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2021Learning to Prompt for Continual Learningcited by 938
  11. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  12. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
28 more
  1. 2021Gradient Projection Memory for Continual Learningcited by 338
  2. 2024LoRA Learns Less and Forgets Lesscited by 313
  3. 2023In-Context Learning Creates Task Vectorscited by 302
  4. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  5. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  6. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  7. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  8. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  9. 2019Episodic Memory in Lifelong Language Learningcited by 188
  10. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
  11. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  12. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
  13. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  14. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  15. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 80
  16. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  17. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  18. 2021Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connectorcited by 34
  19. 2023Continual Learning with Scaled Gradient Projectioncited by 33
  20. 2023Task Arithmetic with LoRA for Continual Learningcited by 31
  21. 2024Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfercited by 31
  22. 2022Balancing Stability and Plasticity through Advanced Null Space in Continual Learningcited by 29
  23. 2023Continual Learning with Low Rank Adaptationcited by 27
  24. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  25. 2024Is Parameter Collision Hindering Continual Learning in LLMs?cited by 15
  26. 2024Learning Attentional Mixture of LoRAs for Language Model Continual Learningcited by 10
  27. 2022Parameter-Efficient Finetuning for Robust Continual Multilingual Learningcited by 9
  28. 2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agentscited by 7
  2. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
  3. 2026Collaborative Parameter Learning: Mitigating Forgetting via Parameter-Level Gradient Analysiscited by 1
  4. 2026SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Modelscited by 1
  5. 2026Make LLM Learn to Synthesize from Streaming Experiences through Feedbackcited by 1
  6. 2026Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repaircited by 0
  7. 2026Rotation-Preserving Supervised Fine-Tuningcited by 0
  8. 2026MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Usecited by 0
  9. 2026Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Modelscited by 0
  10. 2026Generalizable Lifelong Model Editing via Preference Optimizationcited by 0