mlsanityData to 7 Oct 2026

Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs

arXiv 2402.08005 ↗First posted 12 Feb 2024cs.CL cs.LGcited by 10

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

In this paper, we introduce refined Direct Preference Optimization (rDPO), a method for improving the behavioral alignment of Large Language Models (LLMs) without the need for human-annotated data. The method involves creating synthetic data using self-critique prompting by a teacher LLM and then utilising a generalized DPO loss function to distil to a student LLM. The loss function incorporates an additional external reward model to improve the quality of synthetic data, making rDPO robust to potential noise in the synthetic dataset. rDPO is shown to be effective in a diverse set of behavioural alignment tasks, such as improved safety, robustness against role-playing, and reduced sycophancy. Code to be released at https://github.com/vicgalle/refined-dpo.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 41 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

36 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
21 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  5. 2024Self-Rewarding Language Modelscited by 654
  6. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  8. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  13. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  14. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  15. 2023System 2 Attention (is something you might need too)cited by 85
  16. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  17. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Modelscited by 14
  18. 2023Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Datacited by 12
  19. 2024GRATH: Gradual Self-Truthifying for Large Language Modelscited by 6
  20. 2023Distilled Self-Critique of LLMs with Synthetic Data: a Bayesian Perspectivecited by 5
  21. 2023ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIFcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 1.

10 papers.

  1. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  2. 2024Constraint Back-translation Improves Complex Instruction Following of Large Language Modelscited by 11
  3. 2024Length Desensitization in Direct Preference Optimizationcited by 8
  4. 2025CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdomcited by 6
  5. 2024Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Interventioncited by 5
  6. 2025TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuningcited by 4
  7. 2025Sycophancy Claims about Language Models: The Missing Human-in-the-Loopcited by 4
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  9. 2024Open Artificial Knowledgecited by 1
  10. 2025PanicToCalm: A Proactive Counseling Agent for Panic Attackscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  2. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  3. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  4. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  5. 2024sDPO: Don't Use Your Data All at Once
  6. 2025ComPO: Preference Alignment via Comparison Oracles
  7. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
  8. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment