# Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editing

arXiv: https://arxiv.org/abs/2506.03490 · HTML: https://mlsanity.com/paper/2506.03490/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 56 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 30 of 36)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors](https://mlsanity.com/paper/2211.11031/) (2022)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models](https://mlsanity.com/paper/2410.02355/) (2024)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use](https://mlsanity.com/paper/1905.05134/) (2019)
- [What do we need to build explainable AI systems for the medical domain?](https://mlsanity.com/paper/1712.09923/) (2017)
- [Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks](https://mlsanity.com/paper/2410.20266/) (2024)
- [Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue](https://mlsanity.com/paper/2401.04700/) (2024)
- [AnyEdit: Edit Any Knowledge Encoded in Language Models](https://mlsanity.com/paper/2502.05628/) (2025)
- [Massive Editing for Large Language Models via Meta Learning](https://mlsanity.com/paper/2311.04661/) (2023)
- [MedExQA: Medical Question Answering Benchmark with Multiple Explanations](https://mlsanity.com/paper/2406.06331/) (2024)
- [The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse](https://mlsanity.com/paper/2402.09656/) (2024)
- [Do Large Language Models Know about Facts?](https://mlsanity.com/paper/2310.05177/) (2023)
- [Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning](https://mlsanity.com/paper/2405.03279/) (2024)
- [Can Knowledge Editing Really Correct Hallucinations?](https://mlsanity.com/paper/2410.16251/) (2024)
- [Emptying the Ocean with a Spoon: Should We Edit Models?](https://mlsanity.com/paper/2310.11958/) (2023)

## Built on by (most cited first, 4 of 4)

- [Fine-tuning Done Right in Model Editing](https://mlsanity.com/paper/2509.22072/) (2025)
- [MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models](https://mlsanity.com/paper/2508.05083/) (2025)
- [MedTextWeaver: Procedural Knowledge Evolution in Agentic Medical Text Editing](https://mlsanity.com/paper/2602.00740/) (2026)
- [MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models](https://mlsanity.com/paper/2609.38543/) (2026)

## Abstract (arXiv)

Recently, knowledge editing (KE) has emerged as a promising approach to update specific facts in Large Language Models (LLMs) without the need for full retraining. Despite the effectiveness in general-domain benchmarks, their applicability to complex medical domain remains largely unexplored. Medical knowledge editing is particularly challenging, as it requires LLMs to internalize the knowledge and generalize to unseen scenarios for effective and interpretable decision-making. In this work, we propose a novel framework called MedEditBench to rigorously evaluate the effectiveness of existing KE methods in the medical domain. In MedEditBench, we introduce a new medical knowledge editing benchmark as well as three different knowledge editing paradigms, which are designed to assess the impact of different knowledge sources for editing. Our findings indicate that current KE methods result in only superficial memorization of the injected information, failing to generalize to new scenarios. To overcome this limitation, we present Self-Generated Rationale Editing (SGR-Edit), which utilizes model-derived rationales as the target knowledge for editing, thereby uncovering the underlying reasoning process and demonstrating significant improvements over existing KE approaches. Additionally, we offer deeper insights into medical knowledge editing, including the localization of medical knowledge in LLMs and the impact of sequential editing on evolving knowledge. This could provide practical guidance for implementing KE methods in real-world medical applications.