mlsanityData to 7 Oct 2026

Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification

arXiv 2601.21244 ↗First posted 29 Jan 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework (LENS), which first prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that LENS significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6× speedup on math reasoning, and a 1.83% gain on scientific and general reasoning. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 42 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

30 of 42 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives6 Oct
  2. 2025ExGRPO: Learning to Reason from Experience2 Oct
  3. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  4. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration19 Aug
  5. 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy6 Aug
  6. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  7. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  8. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  9. 2025Learning to Reason under Off-Policy Guidance21 Apr
  10. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  11. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  12. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
4 more
  1. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 14

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  6. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  7. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  8. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  9. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 67
  10. 2019Dealing with Sparse Rewards in Reinforcement Learningcited by 40
  11. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  12. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actionscited by 15
2 more
  1. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
  2. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  2. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards
  3. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward
  4. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning
  5. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  6. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  7. 2026DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards
  8. 2026SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions