mlsanityData to 7 Oct 2026

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

arXiv 2401.01335 ↗First posted 2 Jan 2024cs.LG cs.AI cs.CLcited by 552

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM's performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at https://github.com/uclaml/SPIN.

Key facts. Cited by 552 papers in the map (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 66 of its 99 references that are in the map. First posted on arXiv 2 Jan 2024.

Builds on

66 of 99 references.

References · 66

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2017Deep reinforcement learning from human preferencescited by 3,846
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023Mistral 7Bcited by 3,065
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2017Improved Training of Wasserstein GANscited by 2,223
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  13. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  14. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  15. 2016Generative Adversarial Imitation Learningcited by 1,576
  16. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  17. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  18. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  19. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  20. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  21. 2023PaLM 2 Technical Reportcited by 1,154
  22. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  23. 2016Least Squares Generative Adversarial Networkscited by 1,000
  24. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  25. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  26. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  27. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  28. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  29. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  30. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  31. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  32. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  33. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  34. 2022Self-critiquing models for assisting human evaluatorscited by 372
  35. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  36. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  37. 2021Curriculum Learning: A Surveycited by 257
  38. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 204; 2025: 189; 2026: 159.

552 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2024Self-Rewarding Language Modelscited by 654
  3. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  5. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  6. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  7. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  8. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  9. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  10. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  11. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  12. 2025TTRL: Test-Time Reinforcement Learningcited by 253
38 more
  1. 2024Iterative Reasoning Preference Optimizationcited by 244
  2. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  3. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  4. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  5. 2025Learning to Reason without External Rewardscited by 199
  6. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  7. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  8. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  9. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  10. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  11. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  12. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 146
  13. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  14. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  15. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  16. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  17. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 121
  18. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  19. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  20. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  21. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  22. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  23. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  24. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  25. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 93
  26. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 93
  27. 2024Learning Dynamics of LLM Finetuningcited by 93
  28. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
  29. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  30. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  31. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 89
  32. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  33. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  34. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  35. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  36. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 85
  37. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  38. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84

50 most cited of 552 All 552 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Response Generation Strategy Selection for Fine-Tuning Large Language Models Through Self-Aligned Perplexity
  2. 2024A Framework for Fine-Tuning LLMs using Heterogeneous Feedback
  3. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  4. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  5. 2024Teaching Language Models to Self-Improve by Learning from Language Feedback
  6. 2026Diversity in Large Language Models under Supervised Fine-Tuning
  7. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Models
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions