mlsanityData to 14 Oct 2026

FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewards

arXiv 2602.00453 ↗First posted 31 Jan 2026cs.LG cs.DCcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has recently emerged as an effective approach for improving the reasoning capabilities of large language models through online multi-objective reinforcement learning. While personalization on private data is increasingly vital, traditional Reinforcement Learning (RL) alignment is often memory-prohibitive for on-device federated learning due to the overhead of maintaining a separate critic network. GRPO's critic-free architecture enables feasible on-device training, yet transitioning to a federated setting introduces systemic challenges: heterogeneous reward definitions, imbalanced multi-objective optimization, and high training costs. We propose FedMOA, a federated GRPO framework for multi-objective alignment under heterogeneous rewards. FedMOA stabilizes local training through an online adaptive weighting mechanism via hypergradient descent, which prioritizes primary reasoning as auxiliary objectives saturate. On the server side, it utilizes a task- and accuracy-aware aggregation strategy to prioritize high-quality updates. Experiments on mathematical reasoning and code generation benchmarks demonstrate that FedMOA consistently outperforms federated averaging, achieving accuracy gains of up to 2.2% while improving global performance, personalization, and multi-objective balance.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 41 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

23 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance18 Aug
  2. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training28 May
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 18

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  6. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  7. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  8. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 107
  9. 2024On Protecting the Data Privacy of Large Language Models (LLMs): A Surveycited by 96
  10. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  11. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 69
  12. 2021Communication Efficiency in Federated Learning: Achievements and Challengescited by 40
6 more
  1. 2023FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Modelscited by 26
  2. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  3. 2023Federated Q-Learning: Linear Regret Speedup with Low Communication Costcited by 19
  4. 2024FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHFcited by 12
  5. 2024On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representationscited by 4
  6. 2025Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devicescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0
  2. 2026Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FGRPO: Federated GRPO with Adaptive Aggregation on Non-IID Data
  2. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  3. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  4. 2025DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning
  5. 2026GRPO Training Dynamics for Small Language Models
  6. 2026MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting
  7. 2026f-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment
  8. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment