mlsanityData to 14 Oct 2026

Distributional Preference Alignment of LLMs via Optimal Transport

arXiv 2406.05882 ↗First posted 9 Jun 2024cs.LG stat.MLcited by 26

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Current LLM alignment techniques use pairwise human preferences at a sample level, and as such, they do not imply an alignment on the distributional level. We propose in this paper Alignment via Optimal Transport (AOT), a novel method for distributional preference alignment of LLMs. AOT aligns LLMs on unpaired preference data by making the reward distribution of the positive samples stochastically dominant in the first order on the distribution of negative samples. We introduce a convex relaxation of this first-order stochastic dominance and cast it as an optimal transport problem with a smooth and convex cost. Thanks to the one-dimensional nature of the resulting optimal transport problem and the convexity of the cost, it has a closed-form solution via sorting on empirical measures. We fine-tune LLMs with this AOT objective, which enables alignment by penalizing the violation of the stochastic dominance of the reward distribution of the positive samples on the reward distribution of the negative samples. We analyze the sample complexity of AOT by considering the dual of the OT problem and show that it converges at the parametric rate. Empirically, we show on a diverse set of alignment datasets and LLMs that AOT leads to state-of-the-art models in the 7B family of models when evaluated with Open LLM Benchmarks and AlpacaEval.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 38 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

19 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 17

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  7. 2018Computational Optimal Transportcited by 780
  8. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  9. 2018Sorting out Lipschitz function approximationcited by 213
  10. 2020Fast Differentiable Sorting and Rankingcited by 197
  11. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  12. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
5 more
  1. 2019Differentiable Ranks and Sorting using Optimal Transportcited by 113
  2. 2024LAB: Large-Scale Alignment for ChatBotscited by 39
  3. 2017An optimal transportation approach for assessing almost stochastic ordercited by 33
  4. 2023Lower Complexity Adaptation for Empirical Entropic Optimal Transportcited by 12
  5. 2022Learning with Stochastic Orderscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 11.

26 papers.

Sort:

By area:

  1. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  2. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
  3. 2025A Survey of Direct Preference Optimizationcited by 36
  4. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 25
  5. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  6. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Userscited by 15
  7. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignmentcited by 10
  8. 2025Hierarchical Refinement: Optimal Transport to Infinity and Beyondcited by 9
  9. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimizationcited by 7
  10. 2024Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Reviewcited by 3
  11. 2025MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignmentcited by 3
  12. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
14 more
  1. 2025ENIGMA: The Geometry of Reasoning and Alignment in Large-Language Modelscited by 2
  2. 2026Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Controlcited by 2
  3. 2026OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Taskscited by 2
  4. 2024Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuningcited by 1
  5. 2025Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraintscited by 1
  6. 2026Efficient Refusal Ablation in LLM through Optimal Transportcited by 1
  7. 2026Linear Social Choice with Few Queries: A Moment-Based Approachcited by 1
  8. 2026AnomSeer: Reinforcing Multimodal LLMs to Reason for Time-Series Anomaly Detectioncited by 0
  9. 2026Transport Clustering: Solving Low-Rank Optimal Transport via Clusteringcited by 0
  10. 2026Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generationcited by 0
  11. 2026Disentangling Optimization Scale from Preference Scale in DPOcited by 0
  12. 2026Understanding Reliability in LLM-based Human Behavior Simulationcited by 0
  13. 2026OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Modelscited by 0
  14. 2026Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Constructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneity
  2. 2024Geometric-Averaged Preference Optimization for Soft Preference Labels
  3. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
  4. 2025Direct Alignment with Heterogeneous Preferences
  5. 2025Why Alignment Must Precede Distillation: A Minimal Working Explanation
  6. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment
  7. 2025Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedback
  8. 2026A Zeroth-Order Paradigm for LLM Preference Alignment