Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning with Real-Time Feedback
On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training
Abstract (arXiv)
Recent works have advanced feedback-based learning systems, whereby a foundation model is able to intake incoming feedback (e.g., from a user) to self-improve, creating a self-loop system of training. However, existing works are limited in needing to consider an offline setup to allow for such feedback-based methods, requiring privileged ground-truth contexts for training. Moreover, there is limited consideration of federated learning (FL), which is particularly well-suited for incorporating external feedback across large networks of end users, for example, but requires methods to be efficient for training on resource-constrained edge devices. Therefore, we introduce SPEAR (Self-Play Enhancement via Advantage-Weighted Refinement), an efficient online learning algorithm for federated LLM fine-tuning. SPEAR utilizes a feedback-guided self-play loop to construct naturally contrastive pairs per prompt which are used to train the model with (i) standard maximum likelihood on satisfactory completions and (ii) confidence-weighted unlikelihood on tail tokens of unsatisfactory completions. SPEAR requires only an interaction-level binary satisfaction signal together with non-answer feedback, without requiring a ground-truth answer to be provided, and does not need expensive group generations, meaning training both online and in a resource-efficient manner is feasible. We validate SPEAR across various benchmark datasets, demonstrating its superior performance in comparison to relevant baselines.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 37 of its 49 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
37 of 49 references.
References · 37
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
25 more
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023Qwen Technical Reportcited by 3,344
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2024Qwen2 Technical Reportcited by 2,120
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
- 2020Adaptive Federated Optimizationcited by 1,046
- 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
- 2019Neural Text Generation with Unlikelihood Trainingcited by 503
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
- 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
- 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 409
- 2026Self-Distillation Enables Continual Learningcited by 247
- 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
- 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 113
- 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
- 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
- 2021Communication Efficiency in Federated Learning: Achievements and Challengescited by 40
- 2024Rethinking Data Selection for Supervised Fine-Tuningcited by 35
- 2024Federated LoRA with Sparse Communicationcited by 26
- 2021Implicit Unlikelihood Training: Improving Neural Text Generation with Reinforcement Learningcited by 16
- 2023A Comparative Evaluation of FedAvg and Per-FedAvg Algorithms for Dirichlet Distributed Heterogeneous Datacited by 14
- 2024Not All Federated Learning Algorithms Are Created Equal: A Performance Evaluation Studycited by 10
- 2025Why Do We Need Warm-up? A Theoretical Perspectivecited by 8
- 2025Cooperative Decentralized Backdoor Attacks on Vertical Federated Learningcited by 3
- 2025TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learningcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs
- 2026Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs
- 2025Towards Understanding Self-play for LLM Reasoning
- 2026Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback
- 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuning
- 2026Online Self-Weighted Fine-Tuning
- 2026Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward
- 2026InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting