# SAKE: Steering Activations for Knowledge Editing

arXiv: https://arxiv.org/abs/2503.01751 · HTML: https://mlsanity.com/paper/2503.01751/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Builds on (most cited first, 30 of 30)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation](https://mlsanity.com/paper/2305.16938/) (2023)
- [How Do Large Language Models Acquire Factual Knowledge During Pretraining?](https://mlsanity.com/paper/2406.11813/) (2024)
- [Controlling Language and Diffusion Models by Transporting Activations](https://mlsanity.com/paper/2410.23054/) (2024)
- [Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering](https://mlsanity.com/paper/2410.15999/) (2024)
- [Representation Surgery: Theory and Practice of Affine Steering](https://mlsanity.com/paper/2402.09631/) (2024)
- [WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing](https://mlsanity.com/paper/2402.10987/) (2024)
- [Uncovering Overfitting in Large Language Model Editing](https://mlsanity.com/paper/2410.07819/) (2024)
- [Evaluating the Reversal Curse in Model Editing](https://mlsanity.com/paper/2310.10322/) (2023)
- [Model Editing by Standard Fine-Tuning](https://mlsanity.com/paper/2402.11078/) (2024)
- [Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings](https://mlsanity.com/paper/2110.02887/) (2021)

## Built on by (most cited first, 13 of 13)

- [EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Models](https://mlsanity.com/paper/2504.15133/) (2025)
- [Steering Language Models Before They Speak: Logit-Level Interventions](https://mlsanity.com/paper/2601.10960/) (2026)
- [Where Steering Signals Come From: Activation Source Selection in Activation Steering](https://mlsanity.com/paper/2607.25270/) (2026)
- [MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs](https://mlsanity.com/paper/2511.07107/) (2025)
- [EtCon: Edit-then-Consolidate for Reliable Knowledge Editing](https://mlsanity.com/paper/2512.04753/) (2025)
- [Targeted Recovery of Weight-Space Mechanisms From Neural Networks](https://mlsanity.com/paper/2607.13047/) (2026)
- [The Anatomy of an Edit: Mechanism-Guided Activation Steering for Knowledge Editing](https://mlsanity.com/paper/2603.20795/) (2026)
- [Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs](https://mlsanity.com/paper/2606.22627/) (2026)
- [Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editing](https://mlsanity.com/paper/2607.01978/) (2026)
- [xMIx: High-Performance Serving-Time Platform for Mechanistic Interpretability Apps](https://mlsanity.com/paper/2607.22595/) (2026)
- [Toward Latent Language Model Skills Steering and Optimization: An Empirical Study](https://mlsanity.com/paper/2608.29459/) (2026)
- [IMPLICIT-Bench: Measuring Implicit Bias in Text-to-Image Models under Neutral Prompts](https://mlsanity.com/paper/2609.24228/) (2026)
- [Do correct answers have stable latent knowledge support?](https://mlsanity.com/paper/2610.15277/) (2026)

## Abstract (arXiv)

As Large Langue Models have been shown to memorize real-world facts, the need to update this knowledge in a controlled and efficient manner arises. Designed with these constraints in mind, Knowledge Editing (KE) approaches propose to alter specific facts in pretrained models. However, they have been shown to suffer from several limitations, including their lack of contextual robustness and their failure to generalize to logical implications related to the fact. To overcome these issues, we propose SAKE, a steering activation method that models a fact to be edited as a distribution rather than a single prompt. Leveraging Optimal Transport, SAKE alters the LLM behavior over a whole fact-related distribution, defined as paraphrases and logical implications. Several numerical experiments demonstrate the effectiveness of this method: SAKE is thus able to perform more robust edits than its existing counterparts.