# Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning

arXiv: https://arxiv.org/abs/2505.24726 · HTML: https://mlsanity.com/paper/2505.24726/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 30 May 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-rewarding alignment and self-training](https://mlsanity.com/synth/self-rewarding-alignment-and-self-training/)

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [Re3: Generating Longer Stories With Recursive Reprompting and Revision](https://mlsanity.com/paper/2210.06774/) (2022)
- [Recursive Introspection: Teaching Language Model Agents How to Self-Improve](https://mlsanity.com/paper/2407.18219/) (2024)
- [Understanding Catastrophic Forgetting in Language Models via Implicit Inference](https://mlsanity.com/paper/2309.10105/) (2023)
- [GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models](https://mlsanity.com/paper/2504.09696/) (2025)
- [Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives](https://mlsanity.com/paper/2401.02009/) (2024)
- [Learning Adaptive Parallel Reasoning with Language Models](https://mlsanity.com/paper/2504.15466/) (2025)
- [Easy Problems That LLMs Get Wrong](https://mlsanity.com/paper/2405.19616/) (2024)
- [LEMMA: Learning from Errors for MatheMatical Advancement in LLMs](https://mlsanity.com/paper/2503.17439/) (2025)
- [Limits for Learning with Language Models](https://mlsanity.com/paper/2306.12213/) (2023)
- [Rethinking Chain-of-Thought from the Perspective of Self-Training](https://mlsanity.com/paper/2412.10827/) (2024)
- [Instruct-of-Reflection: Enhancing Large Language Models Iterative Reflection Capabilities via Dynamic-Meta Instruction](https://mlsanity.com/paper/2503.00902/) (2025)

## Built on by (most cited first, 30 of 33)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026)
- [Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning](https://mlsanity.com/paper/2512.19920/) (2025)
- [Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoning](https://mlsanity.com/paper/2509.23808/) (2025)
- [Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention](https://mlsanity.com/paper/2602.03338/) (2026)
- [Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning](https://mlsanity.com/paper/2509.26578/) (2025)
- [Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts](https://mlsanity.com/paper/2509.21743/) (2025)
- [R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification](https://mlsanity.com/paper/2601.03715/) (2026)
- [SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models](https://mlsanity.com/paper/2510.06871/) (2025)
- [Retrospective Progress-Aware Self-Refinement for LLM Agent Training](https://mlsanity.com/paper/2606.14302/) (2026)
- [Self-Improving World Modelling with Latent Actions](https://mlsanity.com/paper/2602.06130/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoning](https://mlsanity.com/paper/2508.21104/) (2025)
- [Self-Reflective Generation at Test Time](https://mlsanity.com/paper/2510.02919/) (2025)
- [Deep Self-Evolving Reasoning](https://mlsanity.com/paper/2510.17498/) (2025)
- [Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry](https://mlsanity.com/paper/2607.03702/) (2026)
- [Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialogue](https://mlsanity.com/paper/2601.18281/) (2026)
- [Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agents](https://mlsanity.com/paper/2602.05810/) (2026)
- [Learning Stateful Predictive Knowledge From Experience](https://mlsanity.com/paper/2607.28638/) (2026)
- [ScRPO: From Errors to Insights](https://mlsanity.com/paper/2511.06065/) (2025)
- [M3-AD: Reflection-aware Multi-modal, Multi-category, and Multi-dimensional Benchmark and Framework for Industrial Anomaly Detection](https://mlsanity.com/paper/2603.00055/) (2026)
- [Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learning](https://mlsanity.com/paper/2603.15434/) (2026)
- [Input-Time Scaling: Adding Noise and Irrelevance into Less-Is-More Drastically Improves Reasoning Performance and Efficiency](https://mlsanity.com/paper/2508.13654/) (2025)
- [Jenius Agent: Towards Experience-Driven Accuracy Optimization in Real-World Scenarios](https://mlsanity.com/paper/2601.01857/) (2026)
- [Position: Introspective Experience from Conversational Environments as a Path to Better Learning](https://mlsanity.com/paper/2602.14910/) (2026)
- [AIPO: Learning to Reason from Active Interaction](https://mlsanity.com/paper/2605.08401/) (2026)
- [Empowering VLMs for Few-Shot Multimodal Time Series Classification via Tailored Agentic Reasoning](https://mlsanity.com/paper/2605.09395/) (2026)
- [ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning](https://mlsanity.com/paper/2606.13316/) (2026)
- [Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakes](https://mlsanity.com/paper/2609.19156/) (2026)

## Abstract (arXiv)

We explore a method for improving the performance of large language models through self-reflection and reinforcement learning. By incentivizing the model to generate better self-reflections when it answers incorrectly, we demonstrate that a model's ability to solve complex, verifiable tasks can be enhanced even when generating synthetic data is infeasible and only binary feedback is available. Our framework operates in two stages: first, upon failing a given task, the model generates a self-reflective commentary analyzing its previous attempt; second, the model is given another attempt at the task with the self-reflection in context. If the subsequent attempt succeeds, the tokens generated during the self-reflection phase are rewarded. Our experimental results show substantial performance gains across a variety of model architectures, as high as 34.7% improvement at math equation writing and 18.1% improvement at function calling. Notably, smaller fine-tuned models (1.5 billion to 7 billion parameters) outperform models in the same family that are 10 times larger. Our novel paradigm is thus an exciting pathway to more useful and reliable language models that can self-improve on challenging tasks with limited external feedback.