# Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment

arXiv: https://arxiv.org/abs/2408.06266 · HTML: https://mlsanity.com/paper/2408.06266/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 54 of its 64 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 54)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)

## Built on by (most cited first, 30 of 46)

- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [LMUnit: Fine-grained Evaluation with Natural Language Unit Tests](https://mlsanity.com/paper/2412.13091/) (2024)
- [We Can't Understand AI Using our Existing Vocabulary](https://mlsanity.com/paper/2502.07586/) (2025)
- [Phi-Ground Tech Report: Advancing Perception in GUI Grounding](https://mlsanity.com/paper/2507.23779/) (2025)
- [Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings](https://mlsanity.com/paper/2503.15620/) (2025)
- [Bayesian scaling laws for in-context learning](https://mlsanity.com/paper/2410.16531/) (2024)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality](https://mlsanity.com/paper/2506.07171/) (2025)
- [MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge](https://mlsanity.com/paper/2507.21183/) (2025)
- [Training Dynamics Impact Post-Training Quantization Robustness](https://mlsanity.com/paper/2510.06213/) (2025)
- [GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking](https://mlsanity.com/paper/2412.14140/) (2024)
- [Aligning Spoken Dialogue Models from User Interactions](https://mlsanity.com/paper/2506.21463/) (2025)
- [Aligning to What? Limits to RLHF Based Alignment](https://mlsanity.com/paper/2503.09025/) (2025)
- [Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization](https://mlsanity.com/paper/2505.18720/) (2025)
- [Post-training for Efficient Communication via Convention Formation](https://mlsanity.com/paper/2508.06482/) (2025)
- [Neologism Learning for Controllability and Self-Verbalization](https://mlsanity.com/paper/2510.08506/) (2025)
- [InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization](https://mlsanity.com/paper/2503.15880/) (2025)
- [The Differences Between Direct Alignment Algorithms are a Blur](https://mlsanity.com/paper/2502.01237/) (2025)
- [MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment](https://mlsanity.com/paper/2505.23634/) (2025)
- [DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization](https://mlsanity.com/paper/2506.14157/) (2025)
- [dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models](https://mlsanity.com/paper/2603.18806/) (2026)
- [RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking](https://mlsanity.com/paper/2409.17458/) (2024)
- [Querying Databases with Function Calling](https://mlsanity.com/paper/2502.00032/) (2025)
- [C2-DPO: Constrained Controlled Direct Preference Optimization](https://mlsanity.com/paper/2502.17507/) (2025)
- [Latent Principle Discovery for Language Model Self-Improvement](https://mlsanity.com/paper/2505.16927/) (2025)
- [Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains](https://mlsanity.com/paper/2602.00603/) (2026)
- [Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization](https://mlsanity.com/paper/2608.31079/) (2026)
- [Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints](https://mlsanity.com/paper/2508.08466/) (2025)
- [Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning](https://mlsanity.com/paper/2510.14459/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are often aligned using contrastive alignment objectives and preference pair datasets. The interaction between model, paired data, and objective makes alignment a complicated procedure, sometimes producing subpar results. We study this and find that (i) preference data gives a better learning signal when the underlying responses are contrastive, and (ii) alignment objectives lead to better performance when they specify more control over the model during training. Based on these insights, we introduce Contrastive Learning from AI Revisions (CLAIR), a data-creation method which leads to more contrastive preference pairs, and Anchored Preference Optimization (APO), a controllable and more stable alignment objective. We align Llama-3-8B-Instruct using various comparable datasets and alignment objectives and measure MixEval-Hard scores, which correlate highly with human judgments. The CLAIR preferences lead to the strongest performance out of all datasets, and APO consistently outperforms less controllable objectives. Our best model, trained on 32K CLAIR preferences with APO, improves Llama-3-8B-Instruct by 7.65%, closing the gap with GPT4-turbo by 45%. Our code is available at https://github.com/ContextualAI/CLAIR_and_APO.