# How LoRA Remembers? A Parametric Memory Law for LLM Finetuning

arXiv: https://arxiv.org/abs/2605.30260 · HTML: https://mlsanity.com/paper/2605.30260/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [Towards Modular LLMs by Building and Reusing a Library of LoRAs](https://mlsanity.com/paper/2405.11157/) (2024)
- [Text-to-LoRA: Instant Transformer Adaption](https://mlsanity.com/paper/2506.06105/) (2025)
- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](https://mlsanity.com/paper/2602.15902/) (2026)
- [Memory3: Language Modeling with Explicit Memory](https://mlsanity.com/paper/2407.01178/) (2024)
- [Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass](https://mlsanity.com/paper/2411.05877/) (2024)
- [Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancement](https://mlsanity.com/paper/2503.23895/) (2025)
- [Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights](https://mlsanity.com/paper/2506.16406/) (2025)
- [How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?](https://mlsanity.com/paper/2502.14502/) (2025)
- [Understanding LoRA as Knowledge Memory: An Empirical Analysis](https://mlsanity.com/paper/2603.01097/) (2026)
- [Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics](https://mlsanity.com/paper/2602.02343/) (2026)
- [Beyond Memorization: The Challenge of Random Memory Access in Language Models](https://mlsanity.com/paper/2403.07805/) (2024)
- [Context Parametrization with Compositional Adapters](https://mlsanity.com/paper/2509.22158/) (2025)
- [DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering](https://mlsanity.com/paper/2510.12251/) (2025)

## Built on by (most cited first, 3 of 3)

- [Metis: Memory Foundation Model](https://mlsanity.com/paper/2607.26760/) (2026)
- [How Much Can a LoRA Adapter Memorize? Measuring Adapter Capacity in Bits](https://mlsanity.com/paper/2607.21351/) (2026)
- [Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety](https://mlsanity.com/paper/2606.27632/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.