mlsanityData to 14 Oct 2026

Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data

arXiv 2609.16532 ↗First posted 15 Sep 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Continued pretraining (CPT) with data augmentation such as paraphrasing can store inside a large language model (LLM) the knowledge of a small source corpus. The stored knowledge, however, is not always retrieved correctly. We study the eliciting side rather than the storing side: we use preference optimization, which learns from pairs of a preferred (chosen) and a dispreferred (rejected) response, so that the model elicits its stored knowledge more accurately. One proposed approach takes the model's own erroneous response as rejected and the gold answer as chosen, so as to suppress the error. When the target knowledge is partially known, however, most of these rejected responses are factually correct. Using direct preference optimization (DPO) then pushes down rejected responses that contain correct knowledge and differ from the chosen answer only in style, such as length and wording. We propose style-debiased DPO (SD-DPO), which scores whether the rejected response of each pair is factually correct, inverts the preference of such pairs, and weights them so that the learning signal due to differences in style cancels out as a whole. We first test whether, on top of EntiGraph, a representative storing-side method that runs CPT on text synthesized from the corpus, our method adds accuracy efficiently. On QuALITY, the reading-comprehension QA benchmark on which EntiGraph was evaluated, SD-DPO exceeds a baseline we CPT on EntiGraph's synthetic data from the same base model and evaluate with the same procedure. The training tokens this requires are a few dozen times fewer than the additional CPT needed for the same gain. For knowledge updating, the main goal of this work, we use AToKE, a knowledge-editing benchmark for facts that change over time. There, SD-DPO reaches an overall accuracy of 0.982 and answers with the new or the old fact according to the queried period.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 32 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

29 of 32 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training11 May
  2. 2025The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features16 Sep
  3. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  4. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence16 Jun
  5. 2024Knowledge Editing in Language Models via Adapted Direct Preference Optimization14 Jun
  6. 2024Filtered Direct Preference Optimization22 Apr
  7. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  8. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  9. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2022Mass-Editing Memory in a Transformercited by 1,001
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  8. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  9. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  10. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  11. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  12. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
7 more
  1. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  2. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  3. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  4. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  5. 2024Synthetic continued pretrainingcited by 60
  6. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 55
  7. 2026PASTA: A Paraphrasing And Self-Training Approach for Knowledge Updating in LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
  2. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
  3. 2026SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization
  4. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
  5. 2026RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization
  6. 2026Mitigating Mismatch within Reference-based Preference Optimization
  7. 2025Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning
  8. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairs