# Unlocking Efficient, Scalable, and Continual Knowledge Editing with Basis-Level Representation Fine-Tuning

arXiv: https://arxiv.org/abs/2503.00306 · HTML: https://mlsanity.com/paper/2503.00306/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 63 references that are in the map.
- First posted on arXiv 1 Mar 2025.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)

## Built on by (most cited first, 19 of 19)

- [UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models](https://mlsanity.com/paper/2505.14679/) (2025)
- [Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stages](https://mlsanity.com/paper/2603.12658/) (2026)
- [Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs](https://mlsanity.com/paper/2511.20892/) (2025)
- [Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models](https://mlsanity.com/paper/2503.18337/) (2025)
- [Sparse Fine-Tuning of Transformers for Generative Tasks](https://mlsanity.com/paper/2507.10855/) (2025)
- [Towards Universal Debiasing for Language Models-based Tabular Data Generation](https://mlsanity.com/paper/2509.16475/) (2025)
- [DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing](https://mlsanity.com/paper/2604.07965/) (2026)
- [BetaEdit: Null-Space Constrained Sequential Model Editing](https://mlsanity.com/paper/2605.09285/) (2026)
- [ADS-Edit: A Multimodal Knowledge Editing Dataset for Autonomous Driving Systems](https://mlsanity.com/paper/2503.20756/) (2025)
- [Latent Knowledge Scalpel: Precise and Massive Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2508.03741/) (2025)
- [In-Context Compositional Learning via Sparse Coding Transformer](https://mlsanity.com/paper/2511.20194/) (2025)
- [On the Limitations of Rank-One Model Editing in Answering Multi-hop Questions](https://mlsanity.com/paper/2601.04600/) (2026)
- [Representation Finetuning for Continual Learning](https://mlsanity.com/paper/2603.11201/) (2026)
- [Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence](https://mlsanity.com/paper/2606.00570/) (2026)
- [Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs](https://mlsanity.com/paper/2606.22627/) (2026)
- [Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editing](https://mlsanity.com/paper/2607.01978/) (2026)
- [A Taxonomy of Cognitive Capability Gaps in Generative and Agentic AI](https://mlsanity.com/paper/2608.02553/) (2026)
- [Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing](https://mlsanity.com/paper/2608.11660/) (2026)
- [Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing](https://mlsanity.com/paper/2610.02772/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved remarkable performance on various natural language tasks. However, they are trained on static corpora and their knowledge can become outdated quickly in the fast-changing world. This motivates the development of knowledge editing methods designed to update certain knowledge in LLMs without changing unrelated others. To make selective edits, previous efforts often sought to update a small amount of parameters in some specific layer(s) of a LLM. Nonetheless, in challenging scenarios, they still fall short in making successful edits while preserving knowledge irrelevant to the updates simultaneously, resulting in a notable editing-locality trade-off. In this work, we question if the trade-offs are caused by the fact that parameter-based updates have a global effect, i.e., edited parameters affect all inputs indiscriminately. In light of this, we explore the feasibility of representation fine-tuning, which applied some linear update to a few representations in a learned subspace, for knowledge editing. While being effective to enhance an LLM's general ability as demonstrated in the previous work, we theoretically show that this linear update imposes a tension in editing-locality trade-off. Subsequently, BaFT is proposed to break the linearity. BaFT computes a weight for each basis that spans a dimension of the subspace based on the input representation. This input-dependent weighting mechanism allows BaFT to manage different types of knowledge in an adaptive way, thereby achieving a better editing-locality trade-off. Experiments on three LLMs with five editing benchmarks in diverse scenarios show the superiority of our method.