mlsanityData to 14 Oct 2026

Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning

arXiv 2505.24726 ↗First posted 30 May 2025cs.CLcited by 33

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

We explore a method for improving the performance of large language models through self-reflection and reinforcement learning. By incentivizing the model to generate better self-reflections when it answers incorrectly, we demonstrate that a model's ability to solve complex, verifiable tasks can be enhanced even when generating synthetic data is infeasible and only binary feedback is available. Our framework operates in two stages: first, upon failing a given task, the model generates a self-reflective commentary analyzing its previous attempt; second, the model is given another attempt at the task with the self-reflection in context. If the subsequent attempt succeeds, the tokens generated during the self-reflection phase are rewarded. Our experimental results show substantial performance gains across a variety of model architectures, as high as 34.7% improvement at math equation writing and 18.1% improvement at function calling. Notably, smaller fine-tuned models (1.5 billion to 7 billion parameters) outperform models in the same family that are 10 times larger. Our novel paradigm is thus an exciting pathway to more useful and reliable language models that can self-improve on challenging tasks with limited external feedback.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 30 of its 43 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

30 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improve25 Jul

Other references · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2016Learning without Forgettingcited by 2,496
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
17 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  3. 2024A Survey on Large Language Models for Code Generationcited by 658
  4. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  5. 2023Language Models can Solve Computer Taskscited by 372
  6. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  7. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  8. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  9. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  10. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Modelscited by 82
  11. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  12. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
  13. 2024Easy Problems That LLMs Get Wrongcited by 30
  14. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 21
  15. 2023Limits for Learning with Language Modelscited by 17
  16. 2024Rethinking Chain-of-Thought from the Perspective of Self-Trainingcited by 10
  17. 2025Instruct-of-Reflection: Enhancing Large Language Models Iterative Reflection Capabilities via Dynamic-Meta Instructioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 22.

33 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  3. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 19
  4. 2025Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learningcited by 18
  5. 2025Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoningcited by 17
  6. 2026Accurate Failure Prediction in Agents Does Not Imply Effective Failure Preventioncited by 15
  7. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoningcited by 14
  8. 2025Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughtscited by 12
  9. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplificationcited by 12
  10. 2025SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Modelscited by 10
  11. 2026Retrospective Progress-Aware Self-Refinement for LLM Agent Trainingcited by 6
  12. 2026Self-Improving World Modelling with Latent Actionscited by 5
21 more
  1. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  2. 2025PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoningcited by 4
  3. 2025Self-Reflective Generation at Test Timecited by 3
  4. 2025Deep Self-Evolving Reasoningcited by 3
  5. 2026Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retrycited by 3
  6. 2026Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialoguecited by 2
  7. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  8. 2026Learning Stateful Predictive Knowledge From Experiencecited by 2
  9. 2025ScRPO: From Errors to Insightscited by 1
  10. 2026M3-AD: Reflection-aware Multi-modal, Multi-category, and Multi-dimensional Benchmark and Framework for Industrial Anomaly Detectioncited by 1
  11. 2026Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learningcited by 1
  12. 2025Input-Time Scaling: Adding Noise and Irrelevance into Less-Is-More Drastically Improves Reasoning Performance and Efficiencycited by 0
  13. 2026Jenius Agent: Towards Experience-Driven Accuracy Optimization in Real-World Scenarioscited by 0
  14. 2026Position: Introspective Experience from Conversational Environments as a Path to Better Learningcited by 0
  15. 2026AIPO: Learning to Reason from Active Interactioncited by 0
  16. 2026Empowering VLMs for Few-Shot Multimodal Time Series Classification via Tailored Agentic Reasoningcited by 0
  17. 2026ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learningcited by 0
  18. 2026Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakescited by 0
  19. 2026Reinforcement Learning with Decomposed Subtaskscited by 0
  20. 2026My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learningcited by 0
  21. 2026Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning
  2. 2025From Emergence to Control: Probing and Modulating Self-Reflection in Language Models
  3. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
  4. 2025Rethinking Reflection in Pre-Training
  5. 2023Self-Refine: Iterative Refinement with Self-Feedback
  6. 2026ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning
  7. 2024Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection
  8. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learners