# GEM-Style Constraints for PEFT with Dual Gradient Projection in LoRA

arXiv: https://arxiv.org/abs/2601.02500 · HTML: https://mlsanity.com/paper/2601.02500/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Builds on (most cited first, 19 of 19)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [OSQP: An Operator Splitting Solver for Quadratic Programs](https://mlsanity.com/paper/1711.08013/) (2017)
- [Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges](https://mlsanity.com/paper/1907.00182/) (2019)
- [Continual Reinforcement Learning in 3D Non-stationary Environments](https://mlsanity.com/paper/1905.10112/) (2019)
- [Gradient Episodic Memory with a Soft Constraint for Continual Learning](https://mlsanity.com/paper/2011.07801/) (2020)
- [Lifelong and Continual Learning Dialogue Systems](https://mlsanity.com/paper/2211.06553/) (2022)
- [Continual Policy Distillation of Reinforcement Learning-based Controllers for Soft Robotic In-Hand Manipulation](https://mlsanity.com/paper/2404.04219/) (2024)

## Built on by (most cited first, 1 of 1)

- [Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning](https://mlsanity.com/paper/2605.08949/) (2026)

## Abstract (arXiv)

Full fine-tuning of Large Language Models (LLMs) is computationally costly, motivating Continual Learning (CL) approaches that utilize parameter-efficient adapters. We revisit Gradient Episodic Memory (GEM) within the Low-Rank Adapter (LoRA) subspace and introduce I-GEM: a fixed-budget, GPU-resident dual projected-gradient approximation to GEM's quadratic projection. By constraining non-interference solely within the adapter parameters, I-GEM preserves GEM-like stability with orders-of-magnitude lower mean projection overhead. On a 3-task AG News split with induced domain drift, using GPT-2 (355M) and LoRA (r=8), I-GEM matches GEM's average accuracy (within ~!0.04 pts) and outperforms A-GEM by ~!1.4 pts. Crucially, it reduces projection time vs. GEM by a factor of ~!103. These results suggest that applying GEM constraints in the LoRA subspace is a practical pathway for continual learning at the LLM scale.