# Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing

arXiv: https://arxiv.org/abs/2408.12456 · HTML: https://mlsanity.com/paper/2408.12456/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 6 of its 27 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 6 of 6)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [Knowledge Graph Enhanced Large Language Model Editing](https://mlsanity.com/paper/2402.13593/) (2024)
- [Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps](https://mlsanity.com/paper/2302.00456/) (2023)
- [Model Editing by Standard Fine-Tuning](https://mlsanity.com/paper/2402.11078/) (2024)
- [InstructEdit: Instruction-based Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2402.16123/) (2024)

## Built on by (most cited first, 9 of 9)

- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [How do Transformers Learn Implicit Reasoning?](https://mlsanity.com/paper/2505.23653/) (2025)
- [Context-Robust Knowledge Editing for Language Models](https://mlsanity.com/paper/2505.23026/) (2025)
- [Keys to Robust Edits: from Theoretical Insights to Practical Advances](https://mlsanity.com/paper/2410.09338/) (2024)
- [LLM Unlearning Should Be Form-Independent](https://mlsanity.com/paper/2506.07795/) (2025)
- [Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models](https://mlsanity.com/paper/2601.03542/) (2026)
- [Localized LoRA: A Structured Low-Rank Approximation for Efficient Fine-Tuning](https://mlsanity.com/paper/2506.00236/) (2025)
- [The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models](https://mlsanity.com/paper/2604.05995/) (2026)
- [The Power of Power Law: Asymmetry Enables Compositional Reasoning](https://mlsanity.com/paper/2604.22951/) (2026)

## Abstract (arXiv)

Large language models (LLMs) face challenges with internal knowledge inaccuracies and outdated information. Knowledge editing has emerged as a pivotal approach to mitigate these issues. Although current knowledge editing techniques exhibit promising performance in single-hop reasoning tasks, they show limitations when applied to multi-hop reasoning. Drawing on cognitive neuroscience and the operational mechanisms of LLMs, we hypothesize that the residual single-hop knowledge after editing causes edited models to revert to their original answers when processing multi-hop questions, thereby undermining their performance in multihop reasoning tasks. To validate this hypothesis, we conduct a series of experiments that empirically confirm our assumptions. Building on the validated hypothesis, we propose a novel knowledge editing method that incorporates a Knowledge Erasure mechanism for Large language model Editing (KELE). Specifically, we design an erasure function for residual knowledge and an injection function for new knowledge. Through joint optimization, we derive the optimal recall vector, which is subsequently utilized within a rank-one editing framework to update the parameters of targeted model layers. Extensive experiments on GPT-J and GPT-2 XL demonstrate that KELE substantially enhances the multi-hop reasoning capability of edited LLMs.