mlsanityData to 14 Oct 2026

Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models

arXiv 2505.07558 ↗First posted 12 May 2025cs.LG cs.CL stat.MLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human preferences is crucial for safe deployment, yet existing methods assume specific preference models like Bradley-Terry model. This assumption leads to statistical inconsistency, where more data doesn't guarantee convergence to true human preferences. To address this critical gap, we introduce a novel alignment method Direct Density Ratio Optimization (DDRO). DDRO directly estimates the density ratio between preferred and unpreferred output distributions, circumventing the need for explicit human preference modeling. We theoretically prove that DDRO is statistically consistent, ensuring convergence to the true preferred distribution as the data size grows, regardless of the underlying preference structure. Experiments demonstrate that DDRO achieves superior performance compared to existing methods on many major benchmarks. DDRO unlocks the potential for truly data-driven alignment, paving the way for more reliable and human-aligned LLMs.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 52 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

26 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Binary Classifier Optimization for Large Language Model Alignment6 Apr
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
11 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  7. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  8. 2020Artificial Intelligence, Values and Alignmentcited by 448
  9. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  10. 2020Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimationcited by 45
  11. 2024Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHFcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Preference Optimization by Estimating the Ratio of the Data Distributioncited by 9
  2. 2025Distillation of Large Language Models via Concrete Score Matchingcited by 5
  3. 2026A Minimum Variance Path Principle for Accurate and Stable Score-Based Density Ratio Estimationcited by 3
  4. 2026Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winnercited by 2
  5. 2026One-Step Score-Based Density Ratio Estimationcited by 1
  6. 2026Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ComPO: Preference Alignment via Comparison Oracles
  2. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  3. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  4. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  5. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  6. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  7. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  8. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs