mlsanityData to 14 Oct 2026

Distributed Direct Preference Optimization

arXiv 2605.20696 ↗First posted 20 May 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference-based reinforcement learning (RL) is a key paradigm for aligning policies with human judgments, yet its theoretical behavior in distributed settings where preference data are fragmented across heterogeneous users remains poorly understood. Direct Preference Optimization (DPO) avoids explicit reward modeling but lacks convergence guarantees under federated and decentralized training, where communication constraints and non-IID preferences fundamentally alter optimization dynamics. We provide the first convergence and time-complexity analysis of DPO in distributed environments. Modeling personalized offline RL with user-specific preference distributions, we characterize the induced global optimization landscape. For federated DPO, we derive convergence rates that quantify the impact of client drift, communication frequency, and preference heterogeneity; for decentralized DPO, we establish convergence over general communication graphs and show how spectral connectivity governs optimization speed and consensus. Empirically, we corroborate our theoretical insights on standard alignment benchmarks, demonstrating that our proposed methods not only enjoy strong theoretical guarantees but also deliver robust and scalable performance in practice. The code base is available here.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 56 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

36 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model22 Apr
  2. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  3. 2024Uncertainty-Penalized Direct Preference Optimization26 Oct
  4. 2024Token-level Direct Preference Optimization18 Apr
  5. 2024Direct Preference Optimization with an Offset16 Feb

Other references · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  5. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 194
  6. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  7. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  8. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  9. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  10. 2017Collaborative Deep Learning in Fixed Topology Networkscited by 60
  11. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 60
  12. 2022On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyondcited by 38
19 more
  1. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  2. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Modelscited by 36
  3. 2024Direct Judgement Preference Optimizationcited by 33
  4. 2021The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communicationcited by 31
  5. 2025The European Union general data protection regulation: what it is and what it meanscited by 31
  6. 2020Momentum-Based Policy Gradient Methodscited by 30
  7. 2025DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimizationcited by 27
  8. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 25
  9. 2023Rating-based Reinforcement Learningcited by 15
  10. 2021Safe Multi-Agent Reinforcement Learning through Decentralized Multiple Control Barrier Functionscited by 10
  11. 2023Accelerating Fair Federated Learning: Adaptive Federated Adamcited by 8
  12. 2021MDPGT: Momentum-based Decentralized Policy Gradient Trackingcited by 6
  13. 2025Enhancing PPO with Trajectory-Aware Hybrid Policiescited by 4
  14. 2024DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Modelscited by 3
  15. 2025Consensus-based Decentralized Multi-agent Reinforcement Learning for Random Access Network Optimizationcited by 3
  16. 2024CAESAR: Enhancing Federated RL in Heterogeneous MDPs through Convergence-Aware Sampling with Screeningcited by 2
  17. 2024Decentralized Semantic Traffic Control in AVs Using RL and DQN for Dynamic Roadblockscited by 2
  18. 2025SafeSwarm: Decentralized Safe RL for the Swarm of Drones Landing in Dense Crowdscited by 2
  19. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignmentcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  2. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  3. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  4. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  5. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
  6. 2024Towards Federated RLHF with Aggregated Client Preference for LLMs
  7. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
  8. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs