mlsanityData to 7 Oct 2026

R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learning

arXiv 2601.19620 ↗First posted 27 Jan 2026cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large reasoning models (LRMs) aim to solve diverse and complex problems through structured reasoning. Recent advances in group-based policy optimization methods have shown promise in enabling stable advantage estimation without reliance on process-level annotations. However, these methods rely on advantage gaps induced by high-quality samples within the same batch, which makes the training process fragile and inefficient when intra-group advantages collapse under challenging tasks. To address these problems, we propose a reinforcement learning mechanism named emphR3 that along three directions: (1) a emphcross-context underlineReplay strategy that maintains the intra-group advantage by recalling valuable examples from historical trajectories of the same query, (2) an emphin-context self-underlineReflection mechanism enabling models to refine outputs by leveraging past failures, and (3) a emphstructural entropy underlineRanking reward, which assigns relative rewards to truncated or failed samples by ranking responses based on token-level entropy patterns, capturing both local exploration and global stability. We implement our method on Deepseek-R1-Distill-Qwen-1.5B and train it on the DeepscaleR-40k in the math domain. Experiments demonstrate our method achieves SoTA performance on several math benchmarks, representing significant improvements and fewer reasoning tokens over the base models. Code and model will be released.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 34 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

26 of 34 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning10 Jul
  2. 2025First Return, Entropy-Eliciting Explore9 Jul
  3. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework27 Jun
  4. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  5. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  7. 2025SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning22 May
  8. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  9. 2025Learning to Reason under Off-Policy Guidance21 Apr
  10. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  11. 2025FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models21 Mar
  12. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't20 Mar
2 more
  1. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 12

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  8. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  9. 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 48
  10. 2025Towards Effective Code-Integrated Reasoningcited by 23
  11. 2025Thinker: Learning to Think Fast and Slowcited by 10
  12. 2025Relative Entropy Pathwise Policy Optimizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026How Mobile World Model Guides GUI Agents?cited by 8
  2. 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agentscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
  2. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  3. 2025Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models
  4. 2025Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards
  5. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning
  6. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  7. 2025Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling
  8. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning