mlsanityData to 7 Oct 2026

Language Model Self-improvement by Reinforcement Learning Contemplation

arXiv 2305.14483 ↗First posted 23 May 2023cs.CL cs.LGcited by 75

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Large Language Models (LLMs) have exhibited remarkable performance across various natural language processing (NLP) tasks. However, fine-tuning these models often necessitates substantial supervision, which can be expensive and time-consuming to obtain. This paper introduces a novel unsupervised method called LanguageModel Self-Improvement by Reinforcement Learning Contemplation (SIRLC) that improves LLMs without reliance on external labels. Our approach is grounded in the observation that it is simpler for language models to assess text quality than to generate text. Building on this insight, SIRLC assigns LLMs dual roles as both student and teacher. As a student, the LLM generates answers to unlabeled questions, while as a teacher, it evaluates the generated text and assigns scores accordingly. The model parameters are updated using reinforcement learning to maximize the evaluation score. We demonstrate that SIRLC can be applied to various NLP tasks, such as reasoning problems, text generation, and machine translation. Our experiments show that SIRLC effectively improves LLM performance without external supervision, resulting in a 5.6% increase in answering accuracy for reasoning tasks and a rise in BERTScore from 0.82 to 0.86 for translation tasks. Furthermore, SIRLC can be applied to models of different sizes, showcasing its broad applicability.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 30 of its 53 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

30 of 53 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
18 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  9. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  10. 2022Large Language Models Can Self-Improvecited by 656
  11. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 462
  12. 2020Evaluation of Text Generation: A Surveycited by 275
  13. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  14. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  15. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  16. 2019Automatic adaptation of object detectors to new domains using self-trainingcited by 66
  17. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  18. 2021Automated News Summarization Using Transformerscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 26; 2025: 32; 2026: 14.

75 papers.

Sort:

By area:

  1. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  5. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  6. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  7. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  8. 2024Rule Based Rewards for Language Model Safetycited by 106
  9. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  10. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  11. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  12. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
38 more
  1. 2025Self-Adapting Language Modelscited by 71
  2. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolutioncited by 58
  3. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  4. 2024RaFe: Ranking Feedback Improves Query Rewriting for RAGcited by 47
  5. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  6. 2025MagCache: Fast Video Generation with Magnitude-Aware Cachecited by 36
  7. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  8. 2025Self-Improving LLM Agents at Test-Timecited by 30
  9. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Modelscited by 28
  10. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
  11. 2023SELF: Self-Evolution with Language Feedbackcited by 24
  12. 2024Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatabilitycited by 22
  13. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Modelcited by 22
  14. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 19
  15. 2025IPO: Your Language Model is Secretly a Preference Classifiercited by 17
  16. 2024Self-Evolved Reward Learning for LLMscited by 16
  17. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  18. 2025Incentivizing LLMs to Self-Verify Their Answerscited by 16
  19. 2024SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generationcited by 15
  20. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  21. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  22. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 12
  23. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  24. 2024Empowering Language Models with Active Inquiry for Deeper Understandingcited by 11
  25. 2025ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflectioncited by 11
  26. 2025Self-Evolving Vision-Language Models for Image Quality Assessment via Voting and Rankingcited by 10
  27. 2024Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competenciescited by 9
  28. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 9
  29. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8
  30. 2025RLTHF: Targeted Human Feedback for LLM Alignmentcited by 7
  31. 2024Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rolloutscited by 6
  32. 2024SELU: Self-Learning Embodied MLLMs in Unknown Environmentscited by 6
  33. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  34. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 6
  35. 2026ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Modelscited by 6
  36. 2024Autonomous Prompt Engineering in Large Language Modelscited by 5
  37. 2025Controlling Large Language Model with Latent Actionscited by 5
  38. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning
  2. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
  3. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
  4. 2023Enabling Language Models to Implicitly Learn Self-Improvement
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  6. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  7. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
  8. 2026Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models