mlsanityData to 14 Oct 2026

Personalized Federated Direct Preference Optimization via LoRA Adapters

arXiv 2610.12885 ↗First posted 9 Oct 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice. We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client-specific preference patterns. We prove that this decomposition replaces the gradient diversity kappa2 in the FedDPO bound with a residual heterogeneity kappa2eff, which is no larger than kappa2 when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 32 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

13 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 11

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  3. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  4. 2021Federated Learning for Smart Healthcare: A Surveycited by 184
  5. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  6. 2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 59
  7. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40
  8. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 38
  9. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 33
  10. 2025FedP2EFT: Federated Learning to Personalize PEFT for Multilingual LLMscited by 2
  11. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignmentcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
  2. 2026Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
  3. 2026Private Direct Preference Optimization for LLM Alignment
  4. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignment
  5. 2024Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models
  6. 2026APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs
  7. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignment
  8. 2024Towards Federated RLHF with Aggregated Client Preference for LLMs