# Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation

arXiv: https://arxiv.org/abs/2607.05541 · HTML: https://mlsanity.com/paper/2607.05541/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 21 of 21)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Certified Robustness to Adversarial Examples with Differential Privacy](https://mlsanity.com/paper/1802.03471/) (2018)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Adversarial Attacks and Defences: A Survey](https://mlsanity.com/paper/1810.00069/) (2018)
- [How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses](https://mlsanity.com/paper/2305.10862/) (2023)
- [Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness](https://mlsanity.com/paper/2002.03239/) (2020)
- [DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems](https://mlsanity.com/paper/1812.10812/) (2018)
- [Scaleable input gradient regularization for adversarial robustness](https://mlsanity.com/paper/1905.11468/) (2019)
- [Detection as Regression: Certified Object Detection by Median Smoothing](https://mlsanity.com/paper/2007.03730/) (2020)
- [MiVOLO: Multi-input Transformer for Age and Gender Estimation](https://mlsanity.com/paper/2307.04616/) (2023)
- [When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks](https://mlsanity.com/paper/2105.06152/) (2021)
- [Higher-Order Certification for Randomized Smoothing](https://mlsanity.com/paper/2010.06651/) (2020)
- [Random Smoothing Might be Unable to Certify ℓ_∞ Robustness for High-Dimensional Images](https://mlsanity.com/paper/2002.03517/) (2020)
- [Approximating Lipschitz continuous functions with GroupSort neural networks](https://mlsanity.com/paper/2006.05254/) (2020)
- [Tight Second-Order Certificates for Randomized Smoothing](https://mlsanity.com/paper/2010.10549/) (2020)
- [Reducing Certified Regression to Certified Classification for General Poisoning Attacks](https://mlsanity.com/paper/2208.13904/) (2022)
- [RS-Reg: Probabilistic and Robust Certified Regression Through Randomized Smoothing](https://mlsanity.com/paper/2405.08892/) (2024)

## Abstract (arXiv)

Reinforcement Learning is commonly used to train large language models using environmental feedback. In applied settings, the environment usually provides sparse or delayed feedback. This makes it difficult for the model to pinpoint which actions in its reasoning led to success or failure. So, learning effectively from these signals is hard because the model must determine how each failure should inform meaningful behavioral corrections in subsequent iterations. We introduce a training framework, Self-Review Reinforcement Learning, that embeds an explicit self-review step into each RL episode. When a first-pass response fails, the model generates a self-review to identify what went wrong, which conditions an improved second attempt. Unlike inference-time reflection approaches, such as Reflexion, the framework optimizes self-review with policy gradients and internalizes improvements into the base policy via selective distillation, ensuring they persist across future episodes. A cross-episode memory keeps successful self-reviews for reuse when encountering similar tasks in future episodes during training. We evaluate SRRL against a standard RLVR baseline using the GRPO optimizer across two language models, Qwen 3-4B and OLMo-3- 7B, on GSM8K benchmark. SRRL consistently outperforms the RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement.