mlsanityData to 7 Oct 2026

Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

arXiv 2605.17342 ↗First posted 17 May 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Standard RLHF relies on transitive scalar rewards, failing to capture the cyclic nature of human preferences. While some approaches like the General Preference Model (GPM) address this, we identify a theoretical limitation: their implicit formulation entangles hierarchy with cyclicity, failing to guarantee dominant solutions. To address this, we propose the Hybrid Reward-Cyclic (HRC) model, which utilizes game-theoretic decomposition to explicitly disentangle preferences into orthogonal transitive (scalar) and cyclic (vector) components. Complementing this, we introduce Dynamic Self-Play Preference Optimization (DSPPO), which treats alignment as a time-varying game to progressively guide the policy toward the Nash equilibrium. Synthetic data experiments further validate HRC's structural superiority in mixed transitive--cyclic settings, where HRC converges faster and achieves higher accuracy than GPM. Experiments on RewardBench 2 demonstrate that HRC consistently improves over both BT and GPM baselines (e.g., +1.23% on Gemma-2B-it). In particular, its superior performance in the Ties domain empirically validates the model's robustness in handling complex, non-strict preferences. Extensive downstream evaluations on AlpacaEval 2.0, Arena-Hard-v0.1, and MT-Bench confirm the efficacy of our framework. Notably, when using Gemma-2B-it as the base preference model, HRC+DSPPO achieves a peak length-controlled win-rate of 44.75% on AlpacaEval 2.0 and 46.8% on Arena-Hard-v0.1, significantly outperforming SPPO baselines trained with BT or GPM. Our code is publicly available at https://github.com/lab-klc/Hybrid-Reward-Cyclic.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 54 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

40 of 54 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Multiplayer Nash Preference Optimization27 Sep
  2. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  3. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback11 Mar
  4. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent24 Feb
  5. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model18 Dec
  6. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  7. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment3 Oct
  8. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  9. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  10. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  11. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  12. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
9 more
  1. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  2. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  3. 2023Nash Learning from Human Feedback1 Dec
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  8. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2021Ethical and social risks of harm from Language Modelscited by 957
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
7 more
  1. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  2. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  3. 2023AI Alignment: A Comprehensive Surveycited by 256
  4. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  5. 2020Real World Games Look Like Spinning Topscited by 66
  6. 2018Re-evaluating Evaluationcited by 47
  7. 2022On the Limitations of Elo: Real-World Games, are Transitive, not Additivecited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
  2. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  3. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
  4. 2025Unregularized Linear Convergence in Zero-Sum Game from Preference Feedback
  5. 2026S-SPPO: Semantic-Calibrated Self-Play Preference Optimization
  6. 2025RSPO: Regularized Self-Play Alignment of Large Language Models
  7. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment
  8. 2026Efficient Exploration for Iterative Nash Preference Optimization