mlsanityData to 7 Oct 2026

Knowledge Editing in Language Models via Adapted Direct Preference Optimization

arXiv 2406.09920 ↗First posted 14 Jun 2024cs.CL cs.AIcited by 17

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) can become outdated over time as they may lack updated world knowledge, leading to factual knowledge errors and gaps. Knowledge Editing (KE) aims to overcome this challenge using weight updates that do not require expensive retraining. We propose treating KE as an LLM alignment problem. Toward this goal, we introduce Knowledge Direct Preference Optimization (KDPO), a variation of the Direct Preference Optimization (DPO) that is more effective for knowledge modifications. Our method is based on an online approach that continually updates the knowledge stored in the model. We use the current knowledge as a negative sample and the new knowledge we want to introduce as a positive sample in a process called DPO. We also use teacher-forcing for negative sample generation and optimize using the positive sample, which helps maintain localized changes. We tested our KE method on various datasets and models, comparing it to several cutting-edge methods, with 100 and 500 sequential edits. Additionally, we conducted an ablation study comparing our method to the standard DPO approach. Our experimental results show that our modified DPO method allows for more refined KE, achieving similar or better performance compared to previous methods.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 28 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

24 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 22

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  12. 2022Mass-Editing Memory in a Transformercited by 976
10 more
  1. 2021Fast Model Editing at Scalecited by 491
  2. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  3. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  4. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  5. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  6. 2023PMET: Precise Model Editing in a Transformercited by 170
  7. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  8. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  9. 2023Combating Misinformation in the Age of LLMs: Opportunities and Challengescited by 94
  10. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 7; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 33
  2. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  3. 2024Can Editing LLMs Inject Harm?cited by 22
  4. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 16
  5. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
  6. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  7. 2025CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learnerscited by 9
  8. 2025Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editingcited by 6
  9. 2024LLM Surgery: Efficient Knowledge Unlearning and Editing in Large Language Modelscited by 4
  10. 2024FineGates: LLMs Finetuning with Compression using Stochastic Gatescited by 3
  11. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 3
  12. 2024TransformLLM: Adapting Large Language Models via LLM-Transformed Reading Comprehension Textcited by 2
5 more
  1. 2025Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worldscited by 2
  2. 2025EtCon: Edit-then-Consolidate for Reliable Knowledge Editingcited by 1
  3. 2026Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsitycited by 1
  4. 2026Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMscited by 0
  5. 2026Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
  2. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  3. 2025Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information
  4. 2025Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoning
  5. 2025Edit Once, Update Everywhere: A Simple Framework for Cross-Lingual Knowledge Synchronization in LLMs
  6. 2025Benchmarking and Rethinking Knowledge Editing for Large Language Models
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2023Retrieval-augmented Multilingual Knowledge Editing