# SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering

arXiv: https://arxiv.org/abs/2401.17809 · HTML: https://mlsanity.com/paper/2401.17809/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Builds on (most cited first, 20 of 20)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Transformer-Patcher: One Mistake worth One Neuron](https://mlsanity.com/paper/2301.09785/) (2023)
- [Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue](https://mlsanity.com/paper/2401.04700/) (2024)
- [Massive Editing for Large Language Models via Meta Learning](https://mlsanity.com/paper/2311.04661/) (2023)
- [Unveiling the Pitfalls of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2310.02129/) (2023)
- [Semantic Structure and Interpretability of Word Embeddings](https://mlsanity.com/paper/1711.00331/) (2017)
- [Interpreting Embedding Spaces by Conceptualization](https://mlsanity.com/paper/2209.00445/) (2022)
- [Bort: Towards Explainable Neural Networks with Bounded Orthogonal Constraint](https://mlsanity.com/paper/2212.09062/) (2022)

## Built on by (most cited first, 7 of 7)

- [Model Editing by Standard Fine-Tuning](https://mlsanity.com/paper/2402.11078/) (2024)
- [Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness](https://mlsanity.com/paper/2506.05735/) (2025)
- [Model Editing for LLMs4Code: How Far are We?](https://mlsanity.com/paper/2411.06638/) (2024)
- [Identifying Knowledge Editing Types in Large Language Models](https://mlsanity.com/paper/2409.19663/) (2024)
- [STAR: Boosting Time Series Foundation Models for Anomaly Detection through State-aware Adapter](https://mlsanity.com/paper/2510.16014/) (2025)
- [MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge](https://mlsanity.com/paper/2603.15117/) (2026)
- [KEditVis: A Visual Analytics System for Knowledge Editing of Large Language Models](https://mlsanity.com/paper/2603.29689/) (2026)

## Abstract (arXiv)

The general capabilities of large language models (LLMs) make them the infrastructure for various AI applications, but updating their inner knowledge requires significant resources. Recent model editing is a promising technique for efficiently updating a small amount of knowledge of LLMs and has attracted much attention. In particular, local editing methods, which directly update model parameters, are proven suitable for updating small amounts of knowledge. Local editing methods update weights by computing least squares closed-form solutions and identify edited knowledge by vector-level matching in inference, which achieve promising results. However, these methods still require a lot of time and resources to complete the computation. Moreover, vector-level matching lacks reliability, and such updates disrupt the original organization of the model's parameters. To address these issues, we propose a detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching and adds them to the subject word embeddings in Transformer input. To get these editing embeddings, we propose optimizing then suppressing fusion method, which first optimizes learnable embedding vectors for the editing target and then suppresses the Knowledge Embedding Dimensions (KEDs) to obtain final editing embeddings. We thus propose SWEAoplusOS method for editing factual knowledge in LLMs. We demonstrate the overall state-of-the-art (SOTA) performance of SWEAoplusOS on the CounterFact and zsRE datasets. To further validate the reasoning ability of SWEAoplusOS in editing knowledge, we evaluate it on the more complex RippleEdits benchmark. The results demonstrate that SWEAoplusOS possesses SOTA reasoning ability.