mlsanityData to 7 Oct 2026

FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Models

arXiv 2511.16992 ↗First posted 21 Nov 2025cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning Large Language Models (LLMs) with human values often involves balancing multiple, conflicting objectives such as helpfulness and harmlessness. Training these models is computationally intensive, and centralizing the process raises significant data privacy concerns. Federated Learning (FL) offers a compelling alternative, but existing Federated Multi-Objective Optimization (FMOO) methods face severe communication bottlenecks as their reliance on transmitting multiple gradients to a server is unscalable for large models. We introduce FIRM (Federated In-client Regularized Multi-objective alignment), a novel algorithm that achieves both client disagreement drift mitigation and communication efficiency. In FIRM, each client locally solves a regularized multi-objective optimization problem. By directly mitigating client disagreement drift through in-client regularization, our method eliminates the need for the multi-gradient transmissions common in prior works. Consequently, clients need only to transmit a single set of adapted parameters, maintaining high communication efficiency. We prove that our algorithm converges to Pareto-stationary points and, to our knowledge, provide the first finite-time convergence guarantees for this federated multi-objective alignment setting. Empirically, we show that FIRM leads to smoother training dynamics, reduced client disagreement drift, and improved reward trade-offs compared to baselines. We further propose a method to incorporate a preference over the objectives and report empirical Pareto plots, demonstrating that FIRM can smoothly adapt trade-offs between objectives in response to specified preferences.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 38 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

24 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models2 Jul
  2. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  9. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  10. 2022ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situationscited by 36
  11. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 29
  12. 2023Integration of Large Language Models and Federated Learningcited by 28
9 more
  1. 2020Reanalysis of Variance Reduced Temporal Difference Learningcited by 23
  2. 2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learningcited by 14
  3. 2023Federated Multi-Objective Learningcited by 11
  4. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Sufficescited by 11
  5. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  6. 2025Provably Efficient RL for Linear MDPs under Instantaneous Safety Constraints in Non-Convex Feature Spacescited by 6
  7. 2024Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learningcited by 4
  8. 2025EMORL: Ensemble Multi-Objective Reinforcement Learning for Efficient and Flexible LLM Fine-Tuningcited by 4
  9. 2024Federated Communication-Efficient Multi-Objective Optimizationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Step Toward Federated Pretraining of Multimodal Large Language Models
  2. 2025Pareto Multi-Objective Alignment for Language Models
  3. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneity
  4. 2026AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments
  5. 2025FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Models
  6. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
  7. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignment
  8. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment