mlsanityData to 13 Oct 2026

Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs

arXiv 2605.09922 ↗First posted 11 May 2026cs.CL cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

While recent self-training approaches have reduced reliance on human-labeled data for aligning LLMs, they still face critical limitations: (i) sensitivity to synthetic data quality, leading to instability and bias amplification in iterative training; (ii) ineffective optimization due to a diminishing gap between positive and negative responses over successive training iterations. In this paper, we propose Team-based self-Play with dual Adaptive Weighting (TPAW), a novel self-play algorithm designed to improve alignment in a fully self-supervised setting. TPAW adopts a team-based framework in which the current policy model both collaborates with and competes against historical checkpoints, promoting more stable and efficient optimization. To further enhance learning, we design two adaptive weighting mechanisms: (i) a response reweighting scheme that adjusts the importance of target responses, and (ii) a player weighting strategy that dynamically modulates each team member's contribution during training. Initialized from a SFT model, TPAW iteratively refines alignment without requiring additional human supervision. Experimental results demonstrate that TPAW consistently outperforms existing baselines across various base models and LLM benchmarks. Our code is publicly available at https://github.com/lab-klc/TPAW.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 24 of its 39 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

24 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Self-Improvement in Language Models: The Sharpening Mechanism2 Dec
  2. 2024CREAM: Consistency Regularized Self-Rewarding Language Models16 Oct
  3. 2024Self-Play Preference Optimization for Language Model Alignment1 May

Other references · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  5. 2017Deep reinforcement learning from human preferencescited by 3,871
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  7. 2020Learning to summarize from human feedbackcited by 2,575
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,289
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 954
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 833
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 617
9 more
  1. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 333
  2. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 259
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  4. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  5. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  6. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  7. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 81
  8. 2024Bias Amplification in Language Model Evolution: An Iterated Learning Perspectivecited by 17
  9. 2025Safety Alignment via Constrained Knowledge Unlearningcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs
  2. 2026Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning with Real-Time Feedback
  3. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combat
  4. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment
  5. 2026InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting
  6. 2024Toward Optimal LLM Alignments Using Two-Player Games
  7. 2026Multi-LLM Collaborative Alignment via Stackelberg Games
  8. 2026PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play