mlsanityData to 7 Oct 2026

Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment

arXiv 2604.04410 ↗First posted 6 Apr 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning language models with human preferences is essential for ensuring their safety and reliability. Although most existing approaches assume specific human preference models such as the Bradley-Terry model, this assumption may fail to accurately capture true human preferences, and consequently, these methods lack statistical consistency, i.e., the guarantee that language models converge to the true human preference as the number of samples increases. In contrast, direct density ratio optimization (DDRO) achieves statistical consistency without assuming any human preference models. DDRO models the density ratio between preferred and non-preferred data distributions using the language model, and then optimizes it via density ratio estimation. However, this density ratio is unstable and often diverges, leading to training instability of DDRO. In this paper, we propose a novel alignment method that is both stable and statistically consistent. Our approach is based on the relative density ratio between the preferred data distribution and a mixture of the preferred and non-preferred data distributions. Our approach is stable since this relative density ratio is bounded above and does not diverge. Moreover, it is statistically consistent and yields significantly tighter convergence guarantees than DDRO. We experimentally show its effectiveness with Qwen 2.5 and Llama 3.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 53 references that are in the map. First posted on arXiv 6 Apr 2026.

Builds on

31 of 53 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Preference Optimization by Estimating the Ratio of the Data Distribution26 May
  2. 2025Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models12 May
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024Binary Classifier Optimization for Large Language Model Alignment6 Apr
  5. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  6. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
12 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  5. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  6. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  7. 2023Nash Learning from Human Feedbackcited by 220
  8. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  9. 2020Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimationcited by 43
  10. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecificationcited by 20
  11. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
  12. 2021Meta-Learning for Relative Density-Ratio Estimationcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ComPO: Preference Alignment via Comparison Oracles
  2. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
  3. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  4. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  5. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  6. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  7. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  8. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier