mlsanityData to 14 Oct 2026

RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization

arXiv 2510.02172 ↗First posted 2 Oct 2025cs.CLcited by 15

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with human-annotated data has boosted chain-of-thought reasoning in large reasoning models, but these gains come at high costs in labeled data while faltering on harder tasks. A natural next step is experience-driven learning, where models improve without curated labels by adapting to unlabeled data. We introduce RESTRAIN (REinforcement learning with Self-restraint), a self-penalizing RL framework that converts the absence of gold labels into a useful learning signal. Instead of overcommitting to spurious majority votes, RESTRAIN exploits signals from the model's entire answer distribution: penalizing overconfident rollouts and low-consistency examples while preserving promising reasoning chains. The self-penalization mechanism integrates seamlessly into policy optimization methods such as GRPO, enabling continual self-improvement without supervision. On challenging reasoning benchmarks, RESTRAIN delivers large gains using only unlabeled data. With Qwen3-4B-Base and OctoThinker Hybrid-8B-Base, it improves Pass@1 by up to +140.7 percent on AIME25, +36.2 percent on MMLU_STEM, and +19.6 percent on GPQA-Diamond, nearly matching gold-label training while using no gold labels. These results demonstrate that RESTRAIN establishes a scalable path toward stronger reasoning without gold labels.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 44 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

39 of 44 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  2. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning20 Jun
  3. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models5 Jun
  4. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  5. 2025Maximizing Confidence Alone Improves Reasoning28 May
  6. 2025Can Large Reasoning Models Self-Train?27 May
  7. 2025Learning to Reason without External Rewards26 May
  8. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  9. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  10. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  12. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
5 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  5. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2024Self-Rewarding Language Modelscited by 681
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  12. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
10 more
  1. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  2. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  3. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  4. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  5. 2025Concise Reasoning via Reinforcement Learningcited by 79
  6. 2024Self-Consistency Preference Optimizationcited by 50
  7. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  8. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 36
  9. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 32
  10. 2022Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 13.

15 papers.

Sort:

By area:

  1. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
  2. 2026Maximum Likelihood Reinforcement Learningcited by 36
  3. 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolutioncited by 12
  4. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoningcited by 10
  5. 2026When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewardscited by 6
  6. 2026Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifierscited by 4
  7. 2026Distribution-Aware Reward Estimation for Test-Time Reinforcement Learningcited by 3
  8. 2026Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoningcited by 3
  9. 2026Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanismcited by 2
  10. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  11. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 2
  12. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1
3 more
  1. 2026Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVRcited by 1
  2. 2026Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolvingcited by 0
  3. 2026Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
  2. 2026Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning
  3. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
  4. 2025Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study
  5. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
  6. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoning
  7. 2025RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning
  8. 2026Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting