mlsanityData to 14 Oct 2026

APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport

arXiv 2510.10963 ↗First posted 13 Oct 2025cs.LG cs.AIcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The reward model (RM) plays a crucial role in aligning Large Language Models (LLMs) with human preferences through Reinforcement Learning, where the Bradley-Terry (BT) objective has been recognized as simple yet powerful, specifically for pairwise preference learning. However, BT-based RMs often struggle to effectively distinguish between similar preference responses, leading to insufficient separation between preferred and non-preferred outputs. Consequently, they may easily overfit easy samples and cannot generalize well to Out-Of-Distribution (OOD) samples, resulting in suboptimal performance. To address these challenges, this paper introduces an effective enhancement to BT-based RMs through an adaptive margin mechanism. Specifically, we design to dynamically adjust the RM focus on more challenging samples through margins, based on both semantic similarity and model-predicted reward differences, which is approached from a distributional perspective solvable with Optimal Transport (OT). By incorporating these factors into a principled OT cost matrix design, our adaptive margin enables the RM to better capture distributional differences between chosen and rejected responses, yielding significant improvements in performance, convergence speed, and generalization capabilities. Experimental results across multiple benchmarks demonstrate that our method outperforms several existing RM techniques, showcasing enhanced performance in both In-Distribution (ID) and OOD settings. Moreover, RLHF experiments support our practical effectiveness in better aligning LLMs with human preferences. Our code is available at https://github.com/BIRlz/APLOT

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 53 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

35 of 53 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment16 Feb
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  3. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  4. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  6. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  7. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  8. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2022Scaling Laws for Reward Model Overoptimization19 Oct

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
13 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  2. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  4. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  5. 2017Sampling Matters in Deep Embedding Learningcited by 285
  6. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  7. 2024PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluationcited by 26
  8. 2022Graph Enhanced Contrastive Learning for Radiology Findings Summarizationcited by 24
  9. 2022Learning to Re-weight Examples with Optimal Transport for Imbalanced Classificationcited by 23
  10. 2025Psyche-R1: Towards Reliable Psychological LLMs through Unified Empathy, Expertise, and Reasoningcited by 20
  11. 2025Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmarkcited by 12
  12. 2025Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigmcited by 6
  13. 2024Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
  2. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5
  3. 2026MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucinationcited by 5
  4. 2026AnomSeer: Reinforcing Multimodal LLMs to Reason for Time-Series Anomaly Detectioncited by 0
  5. 2026VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inferencecited by 0
  6. 2026From Soft Targets to Reward Signals: How Assignment and Reward Objectives Interactcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
  2. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
  3. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  4. 2026Bayesian Preference Learning for Test-Time Steerable Reward Models
  5. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives
  6. 2025On the Robustness of Reward Models for Language Model Alignment
  7. 2026In-Context Reward Adaptation for Robust Preference Modeling
  8. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillation