# How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?

arXiv: https://arxiv.org/abs/2502.14502 · HTML: https://mlsanity.com/paper/2502.14502/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 24 of 24)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning](https://mlsanity.com/paper/2010.15875/) (2020)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [Physics of Language Models: Part 3.2, Knowledge Manipulation](https://mlsanity.com/paper/2309.14402/) (2023)
- [Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge](https://mlsanity.com/paper/2102.03315/) (2021)
- [Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark](https://mlsanity.com/paper/2305.17553/) (2023)
- [A Comprehensive Survey on Data Augmentation](https://mlsanity.com/paper/2405.09591/) (2024)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts](https://mlsanity.com/paper/2405.06524/) (2024)
- [Deciphering the Interplay of Parametric and Non-parametric Memory in Retrieval-augmented Language Models](https://mlsanity.com/paper/2410.05162/) (2024)
- [DeepPavlov at SemEval-2024 Task 8: Leveraging Transfer Learning for Detecting Boundaries of Machine-Generated Texts](https://mlsanity.com/paper/2405.10629/) (2024)

## Built on by (most cited first, 13 of 13)

- [Understanding LoRA as Knowledge Memory: An Empirical Analysis](https://mlsanity.com/paper/2603.01097/) (2026)
- [Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA](https://mlsanity.com/paper/2505.21115/) (2025)
- [How LoRA Remembers? A Parametric Memory Law for LLM Finetuning](https://mlsanity.com/paper/2605.30260/) (2026)
- [Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models](https://mlsanity.com/paper/2607.12962/) (2026)
- [From Memorization to Generalization: Fine-Tuning Large Language Models for Biomedical Term-to-Identifier Normalization](https://mlsanity.com/paper/2510.19036/) (2025)
- [How Much Can a LoRA Adapter Memorize? Measuring Adapter Capacity in Bits](https://mlsanity.com/paper/2607.21351/) (2026)
- [Latent Knowledge as a Predictor of Fact Acquisition in Fine-Tuned Large Language Models](https://mlsanity.com/paper/2601.18468/) (2026)
- [daVinci-LLM:Towards the Science of Pretraining](https://mlsanity.com/paper/2603.27164/) (2026)
- [ChipLingo: A Systematic Training Framework for Large Language Models in EDA](https://mlsanity.com/paper/2604.27415/) (2026)
- [Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation](https://mlsanity.com/paper/2605.07111/) (2026)
- [PALoRA: Projection-Adaptive LoRA for Preserving Reasoning in Large Language Models](https://mlsanity.com/paper/2605.24549/) (2026)
- [SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies](https://mlsanity.com/paper/2607.25716/) (2026)
- [SemiAdapt-Instruct: Extensible Instruction Tuning via Latent Domain-Specialised Adapters](https://mlsanity.com/paper/2608.05161/) (2026)

## Abstract (arXiv)

The performance of Large Language Models (LLMs) on many tasks is greatly limited by the knowledge learned during pre-training and stored in the model's parameters. Low-rank adaptation (LoRA) is a popular and efficient training technique for updating or domain-specific adaptation of LLMs. In this study, we investigate how new facts can be incorporated into the LLM using LoRA without compromising the previously learned knowledge. We fine-tuned Llama-3.1-8B-instruct using LoRA with varying amounts of new knowledge. Our experiments have shown that the best results are obtained when the training data contains a mixture of known and new facts. However, this approach is still potentially harmful because the model's performance on external question-answering benchmarks declines after such fine-tuning. When the training data is biased towards certain entities, the model tends to regress to few overrepresented answers. In addition, we found that the model becomes more confident and refuses to provide an answer in only few cases. These findings highlight the potential pitfalls of LoRA-based LLM updates and underscore the importance of training data composition and tuning parameters to balance new knowledge integration and general model capabilities.