Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Post-alignment of large language models (LLMs) is critical in improving their utility, safety, and alignment with human intentions. Direct preference optimisation (DPO) has become one of the most widely used algorithms for achieving this alignment, given its ability to optimise models based on human feedback directly. However, the vast number of DPO variants in the literature has made it increasingly difficult for researchers to navigate and fully grasp the connections between these approaches. This paper introduces a unifying framework inspired by mutual information, which proposes a new loss function with flexible priors. By carefully specifying these priors, we demonstrate that many existing algorithms, such as SimPO, TDPO, SparsePO, and others, can be derived from our framework. This unification offers a clearer and more structured approach, allowing researchers to understand the relationships between different DPO variants better. We aim to simplify the landscape of DPO algorithms, making it easier for the research community to gain insights and foster further advancements in LLM alignment. Ultimately, we hope our framework can be a foundation for developing more robust and interpretable alignment techniques.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. First posted on arXiv 2 Jan 2025.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 4.
4 papers.
- 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
- 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
- 2025The Differences Between Direct Alignment Algorithms are a Blurcited by 3
- 2025Rethinking the Understanding Ability across LLMs through Mutual Informationcited by 1
- 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO29 May 2025
- 2025Rethinking the Understanding Ability across LLMs through Mutual Information25 May 2025
- 2025The Differences Between Direct Alignment Algorithms are a Blur3 Feb 2025
- 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time3 Feb 2025
Similar, not cited
Close in wording, no citation either way
- 2024sDPO: Don't Use Your Data All at Once
- 2025Less is More: Improving LLM Alignment via Preference Data Selection
- 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
- 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
- 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment
- 2025Provably Convergent Primal-Dual DPO for Constrained LLM Alignment
- 2025ComPO: Preference Alignment via Comparison Oracles
- 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization