# Improving Sparse Memory Finetuning

arXiv: https://arxiv.org/abs/2604.05248 · HTML: https://mlsanity.com/paper/2604.05248/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 9 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Builds on (most cited first, 7 of 7)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025)
- [Memory Layers at Scale](https://mlsanity.com/paper/2412.09764/) (2024)
- [Continual Learning via Sparse Memory Finetuning](https://mlsanity.com/paper/2510.15103/) (2025)
- [SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in LLMs](https://mlsanity.com/paper/2502.02909/) (2025)

## Built on by (most cited first, 1 of 1)

- [FSFM: A Biologically-Inspired Framework for Selective Forgetting of Agent Memory](https://mlsanity.com/paper/2604.20300/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are typically static after training, yet real-world applications require continual adaptation to new knowledge without degrading existing capabilities. Standard approaches to updating models, like full finetuning or parameter-efficient methods (e.g., LoRA), face a fundamental trade-off: catastrophic forgetting. They modify shared dense representations, causing interference across tasks. Sparse Memory Finetuning (SMF) offers a promising alternative by localizing updates to a small subset of parameters in explicit memory layers. In this work, we present an open-source pipeline to retrofit existing pretrained models (Qwen-2.5-0.5B) with sparse memory modules, enabling effective continual learning on consumer hardware. We extend prior work by introducing a theoretically grounded slot-selection mechanism based on Kullback-Leibler (KL) divergence, which prioritizes memory updates for informationally "surprising" tokens relative to a background distribution. Our experiments demonstrate that our retrofitted models can acquire new factual knowledge with minimal forgetting of held-out capabilities, validating the sparse update hypothesis in a practical setting.