# Resolving Lexical Bias in Model Editing

arXiv: https://arxiv.org/abs/2408.10411 · HTML: https://mlsanity.com/paper/2408.10411/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Builds on (most cited first, 30 of 30)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [Transformer-Patcher: One Mistake worth One Neuron](https://mlsanity.com/paper/2301.09785/) (2023)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models](https://mlsanity.com/paper/2308.07269/) (2023)
- [BadEdit: Backdooring large language models by model editing](https://mlsanity.com/paper/2403.13355/) (2024)
- [Model Editing at Scale leads to Gradual and Catastrophic Forgetting](https://mlsanity.com/paper/2401.07453/) (2024)
- [MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA](https://mlsanity.com/paper/2312.11795/) (2023)
- [A Unified Framework for Model Editing](https://mlsanity.com/paper/2403.14236/) (2024)
- [SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2406.11171/) (2024)
- [Towards Robustifying NLI Models Against Lexical Dataset Biases](https://mlsanity.com/paper/2005.04732/) (2020)
- [Perturbation-Restrained Sequential Model Editing](https://mlsanity.com/paper/2405.16821/) (2024)
- [Long-form evaluation of model editing](https://mlsanity.com/paper/2402.09394/) (2024)
- [Consecutive Batch Model Editing with HooK Layers](https://mlsanity.com/paper/2403.05330/) (2024)

## Built on by (most cited first, 4 of 4)

- [Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution](https://mlsanity.com/paper/2502.06809/) (2025)
- [On the Persistent Effects of Lexicality in Large Language Models](https://mlsanity.com/paper/2606.02750/) (2026)
- [Matching Meaning at Scale: Evaluating Semantic Search for 18th-Century Intellectual History through the Case of Locke](https://mlsanity.com/paper/2605.09236/) (2026)
- [Values as Style: Disentangling Values from Semantics with One-Way Mixing for Low-Damage LLM Steering](https://mlsanity.com/paper/2609.39701/) (2026)

## Abstract (arXiv)

Model editing aims to modify the outputs of large language models after they are trained. Previous approaches have often involved direct alterations to model weights, which can result in model degradation. Recent techniques avoid making modifications to the model's weights by using an adapter that applies edits to the model when triggered by semantic similarity in the representation space. We demonstrate that current adapter methods are critically vulnerable to strong lexical biases, leading to issues such as applying edits to irrelevant prompts with overlapping words. This paper presents a principled approach to learning a disentangled representation space that facilitates precise localization of edits by maintaining distance between irrelevant prompts while preserving proximity among paraphrases. In our empirical study, we show that our method (Projector Editor Networks for Model Editing - PENME) achieves state-of-the-art model editing results while being more computationally efficient during inference than previous methods and adaptable across different architectures.