mlsanityData to 7 Oct 2026

PLOT: Enhancing Preference Learning via Optimal Transport

arXiv 2604.01837 ↗First posted 2 Apr 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference learning in Large Language Models (LLMs) has advanced significantly, yet existing methods remain limited by modest performance gains, high computational costs, hyperparameter sensitivity, and insufficient modeling of global token-level relationships. We introduce PLOT, which enhances Preference Learning in fine-tuning-based alignment through a token-level loss derived from Optimal Transport. By formulating preference learning as an Optimal Transport Problem, PLOT aligns model outputs with human preferences while preserving the original distribution of LLMs, ensuring stability and robustness. Furthermore, PLOT leverages token embeddings to capture semantic relationships, enabling globally informed optimization. Experiments across two preference categories - Human Values and Logic & Problem Solving - spanning seven subpreferences demonstrate that PLOT consistently improves alignment performance while maintaining fluency and coherence. These results substantiate optimal transport as a principled methodology for preference learning, establishing a theoretically grounded framework that provides new insights for preference learning of LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 44 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

31 of 44 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  2. 2024D2PO: Discriminator-Guided DPO with Response Evaluation Models2 May
  3. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  4. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  5. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment7 Nov
  6. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  7. 2023Preference Ranking Optimization for Human Alignment30 Jun
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
11 more
  1. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  4. 2020Artificial Intelligence, Values and Alignmentcited by 443
  5. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  6. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  7. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  8. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  9. 2021Alignment of Language Agentscited by 167
  10. 2021The Challenge of Value Alignment: from Fairer Algorithms to AI Safetycited by 34
  11. 2024Length Controlled Generation for Black-box LLMscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  3. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  4. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
  5. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  6. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  7. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
  8. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignment