mlsanityData to 7 Oct 2026

Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment

arXiv 2408.06266 ↗First posted 12 Aug 2024cs.LG cs.AI cs.CLcited by 46

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) are often aligned using contrastive alignment objectives and preference pair datasets. The interaction between model, paired data, and objective makes alignment a complicated procedure, sometimes producing subpar results. We study this and find that (i) preference data gives a better learning signal when the underlying responses are contrastive, and (ii) alignment objectives lead to better performance when they specify more control over the model during training. Based on these insights, we introduce Contrastive Learning from AI Revisions (CLAIR), a data-creation method which leads to more contrastive preference pairs, and Anchored Preference Optimization (APO), a controllable and more stable alignment objective. We align Llama-3-8B-Instruct using various comparable datasets and alignment objectives and measure MixEval-Hard scores, which correlate highly with human judgments. The CLAIR preferences lead to the strongest performance out of all datasets, and APO consistently outperforms less controllable objectives. Our best model, trained on 32K CLAIR preferences with APO, improves Llama-3-8B-Instruct by 7.65%, closing the gap with GPT4-turbo by 45%. Our code is available at https://github.com/ContextualAI/CLAIR_and_APO.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 64 references that are in the map. First posted on arXiv 12 Aug 2024.

Builds on

54 of 64 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  2. 2024PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning25 Jun
  3. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignment29 May
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  6. 2024Binary Classifier Optimization for Large Language Model Alignment6 Apr
  7. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective6 Apr
  8. 2024sDPO: Don't Use Your Data All at Once28 Mar
  9. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  10. 2024Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization6 Mar
  11. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  12. 2024Direct Preference Optimization with an Offset16 Feb
4 more
  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Verbosity Bias in Preference Labeling by Large Language Models16 Oct
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
26 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  13. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  14. 2024Self-Rewarding Language Modelscited by 654
  15. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  16. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  17. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  18. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  19. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  20. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  21. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  22. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  23. 2024Reinforcement Learning for LLM Post-Training: A Surveycited by 91
  24. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  25. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  26. 2024Self-Directed Synthetic Dialogues and Revisions Technical Reportcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 25; 2026: 16.

46 papers.

Sort:

By area:

  1. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  2. 2025A Survey of Direct Preference Optimizationcited by 35
  3. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  4. 2025We Can't Understand AI Using our Existing Vocabularycited by 25
  5. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  6. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  7. 2024Bayesian scaling laws for in-context learningcited by 18
  8. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  9. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
  10. 2025RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimalitycited by 14
  11. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
  12. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 13
34 more
  1. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  2. 2025Aligning to What? Limits to RLHF Based Alignmentcited by 8
  3. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimizationcited by 7
  4. 2025Post-training for Efficient Communication via Convention Formationcited by 7
  5. 2025Neologism Learning for Controllability and Self-Verbalizationcited by 5
  6. 2025The Differences Between Direct Alignment Algorithms are a Blurcited by 3
  7. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimizationcited by 3
  8. 2025MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignmentcited by 3
  9. 2025DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimizationcited by 3
  10. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3
  11. 2024RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreakingcited by 2
  12. 2025Querying Databases with Function Callingcited by 2
  13. 2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2
  14. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  15. 2025Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraintscited by 1
  16. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  17. 2025Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spanscited by 1
  18. 2026Mitigating Mismatch within Reference-based Preference Optimizationcited by 1
  19. 2026Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimizationcited by 1
  20. 2025Latent Principle Discovery for Language Model Self-Improvementcited by 0
  21. 2025BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMscited by 0
  22. 2025Neologism Learning as a Parameter-Efficient Alternative to Fine-Tuning for Model Steeringcited by 0
  23. 2026Displacement-Resistant Extensions of DPO with Nonconvex f-Divergencescited by 0
  24. 2026EngGPT2: Sovereign, Efficient and Open Intelligencecited by 0
  25. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Datacited by 0
  26. 2026Towards Reward Modeling for AI Tutors in Math Mistake Remediationcited by 0
  27. 2026Pref-CTRL: Preference Driven LLM Alignment using Representation Editingcited by 0
  28. 2026Learning Selective LLM Autonomy from Copilot Feedback in Enterprise Customer Support Workflowscited by 0
  29. 2026Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generationcited by 0
  30. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  31. 2026TD-DPO: Difference-Aware Preference Optimization for Mitigating Sycophancy in Clinical Autism Intervention Dialoguecited by 0
  32. 2026SafeBranch: Branch-Pair Safety Alignment for Embodied Agentscited by 0
  33. 2026FiMI Banking: A Sovereign Model for Indian Retail Bankingcited by 0
  34. 2026StalePO: Anchored Token-Level Preference Optimization using Legacy Post-Edits in Machine Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  2. 2025ComPO: Preference Alignment via Comparison Oracles
  3. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  4. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  5. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  6. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards
  7. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  8. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time