# RLSR: Reinforcement Learning from Self Reward

arXiv: https://arxiv.org/abs/2505.08827 · HTML: https://mlsanity.com/paper/2505.08827/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 10 references that are in the map.
- First posted on arXiv 12 May 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 9 of 9)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018)
- [LADDER: Self-Improving LLMs Through Recursive Problem Decomposition](https://mlsanity.com/paper/2503.00735/) (2025)
- [Brief analysis of DeepSeek R1 and its implications for Generative AI](https://mlsanity.com/paper/2502.02523/) (2025)

## Built on by (most cited first, 14 of 14)

- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models](https://mlsanity.com/paper/2509.23863/) (2025)
- [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026)
- [Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learning](https://mlsanity.com/paper/2509.23285/) (2025)
- [More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges](https://mlsanity.com/paper/2607.05904/) (2026)
- [Likelihood-Based Reward Designs for General LLM Reasoning](https://mlsanity.com/paper/2602.03979/) (2026)
- [ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models](https://mlsanity.com/paper/2601.15812/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL](https://mlsanity.com/paper/2605.22217/) (2026)
- [DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models](https://mlsanity.com/paper/2601.09981/) (2026)
- [RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges](https://mlsanity.com/paper/2608.24231/) (2026)
- [IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning](https://mlsanity.com/paper/2605.23997/) (2026)
- [Rewarding Better Thinking for LLM Preference Alignment](https://mlsanity.com/paper/2607.19824/) (2026)

## Abstract (arXiv)

Large language models can generate solutions to complex problems, but training them with reinforcement learning typically requires verifiable rewards that are expensive to create and not possible for all domains. We demonstrate that LLMs can effectively self-improve through self-judging without reference solutions, leveraging the inherent asymmetry between generating and verifying solutions. Our experiments show that models can provide reliable reward signals without ground truth answers, enabling reinforcement learning in domains where verifiable rewards are impractical. By implementing self-judging across Countdown puzzles and integration problems, we achieve performance comparable to formal verification without ground truth solutions. Most notably, Qwen 2.5 7B DeepSeek Distilled trained with self-rewards qualifies for the prestigious MIT Integration Bee competition, performance through self-supervised improvement. When combined with synthetic question generation, we establish a complete self-improvement loop where models generate practice problems, solve them, and evaluate their own performance without any external validation. Our findings demonstrate that LLM judges can provide effective reward signals for training, unlocking reinforcement learning in countless domains previously limited by reward engineering challenges. This work represents a significant step toward autonomous AI systems that continuously improve through self-directed learning rather than human-guided training, potentially accelerating progress across domains where training data is scarce or evaluation is complex.