# From Fact Overwriting to Knowledge Evolution: Causal Editing via On-Policy Self-Distillation

arXiv: https://arxiv.org/abs/2605.28303 · HTML: https://mlsanity.com/paper/2605.28303/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 26 of 26)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [Model Editing at Scale leads to Gradual and Catastrophic Forgetting](https://mlsanity.com/paper/2401.07453/) (2024)
- [A Unified Framework for Model Editing](https://mlsanity.com/paper/2403.14236/) (2024)
- [The Mirage of Model Editing: Revisiting Evaluation in the Wild](https://mlsanity.com/paper/2502.11177/) (2025)
- [Long-form evaluation of model editing](https://mlsanity.com/paper/2402.09394/) (2024)
- [Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing](https://mlsanity.com/paper/2410.17194/) (2024)
- [Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales](https://mlsanity.com/paper/2509.23574/) (2025)
- [Is Model Editing Built on Sand? Revealing Its Illusory Success and Fragile Foundation](https://mlsanity.com/paper/2510.00625/) (2025)
- [Layerwise Recall and the Geometry of Interwoven Knowledge in LLMs](https://mlsanity.com/paper/2502.10871/) (2025)
- [MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality](https://mlsanity.com/paper/2503.02701/) (2025)
- [Revealing the Deceptiveness of Knowledge Editing: A Mechanistic Analysis of Superficial Editing](https://mlsanity.com/paper/2505.12636/) (2025)
- [EtCon: Edit-then-Consolidate for Reliable Knowledge Editing](https://mlsanity.com/paper/2512.04753/) (2025)

## Built on by (most cited first, 1 of 1)

- [Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing](https://mlsanity.com/paper/2608.11660/) (2026)

## Abstract (arXiv)

While Knowledge Editing (KE) enables efficient updates, its dominant Static Fact Overwriting paradigm treats LLMs as discrete databases, forcibly injecting isolated facts. Fracturing pre-trained logical topologies, this triggers Epistemic Dissonance -- a pathology where un-evolved legacy priors force the model to explicitly negate the injected update. Idealized interventions reveal that this is an inherent structural flaw rather than mere algorithmic noise, with a zero-distortion proxy yielding a catastrophic 95.6% self-refutation rate. Given the causally driven nature of real-world knowledge, grounding updates in explicit causal narratives effectively collapses this conflict rate to just 6.6%, underscoring the imperative for a paradigm shift toward Causal Editing. To internalize this evolution, we propose CODE (Causal On-policy Distillation for Editing). By coupling causal bootstrapping with asymmetric on-policy distillation, CODE engraves causal transition logic directly into parametric memory. Experiments on LLaMA-3.1 and Qwen-2.5 show CODE drastically suppresses self-refutation to 1.8% while securing robust multi-hop accuracy (up to 83.5%), seamlessly transforming discrete fact injection into coherent knowledge evolution. Code is available at https://github.com/CrashBugger/CODE.