mlsanityData to 14 Oct 2026

No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping

arXiv 2509.21880 ↗First posted 26 Sep 2025cs.CL cs.AI cs.LGcited by 50

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful framework for improving the reasoning abilities of Large Language Models (LLMs). However, current methods such as GRPO rely only on problems where the model responses to the same input differ in correctness, while ignoring those where all responses receive the same reward -- so-called zero-variance prompts. In this work, we argue that such prompts are not useless but can, in fact, provide meaningful feedback for policy optimization. To this end, we introduce Reinforcement Learning with Zero-Variance Prompts (RL-ZVP), a novel algorithm that extract learning signals from zero-variance prompts. RL-ZVP directly rewards correctness and penalizes errors even without contrasting responses, modulating feedback with token-level characteristics to preserve informative, nuanced signals. Across six math reasoning benchmarks, RL-ZVP achieves significant improvements of up to 8.61 points in accuracy and 7.77 points in pass rate over GRPO, while consistently outperforming other baselines that filter out zero-variance prompts. These results highlight the untapped potential of learning from zero-variance prompts in RLVR. The project page is available at https://bltnynk.github.io/publications/rl-zvp/.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 42 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

34 of 42 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning11 Aug
  2. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning4 Aug
  3. 2025Revisiting LLM Reasoning via Information Bottleneck24 Jul
  4. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  5. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  6. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  7. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  9. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  10. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
4 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2023Let's Verify Step by Step31 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
6 more
  1. 2025s1: Simple test-time scalingcited by 1,341
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  4. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 153
  5. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  6. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 45.

50 papers.

Sort:

By area:

  1. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 42
  2. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  3. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergencecited by 13
  4. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 12
  5. 2026Learning to Hint for Reinforcement Learningcited by 12
  6. 2026Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidancecited by 11
  7. 2025SuperFlow: Training Flow Matching Models with RL on the Flycited by 10
  8. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 10
  9. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoningcited by 9
  10. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Taskscited by 9
  11. 2026Orchard: An Open-Source Agentic Modeling Frameworkcited by 8
  12. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
38 more
  1. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 6
  2. 2026EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimizationcited by 5
  3. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  4. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4
  5. 2026Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewardscited by 4
  6. 2026CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPOcited by 3
  7. 2026Effective Reinforcement Learning for Agentic Search by Recycling Zero-Variance Queries During Trainingcited by 3
  8. 2025Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoEcited by 2
  9. 2026Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewardscited by 2
  10. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2
  11. 2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Trainingcited by 2
  12. 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Searchcited by 2
  13. 2026RLVP: Penalize the Path, Reward the Outcomecited by 2
  14. 2026Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groupscited by 2
  15. 2026MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Modelscited by 1
  16. 2026Nonsense Helps: Prompt Space Perturbation Broadens Reasoning Explorationcited by 1
  17. 2026Selective Off-Policy Reference Tuning with Plan Guidancecited by 1
  18. 2026CuSearch: Curriculum Rollout Sampling via Search Depth for Agentic RAGcited by 1
  19. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMscited by 1
  20. 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1
  21. 2026Drowning in Routine: Signal Dilution in Multi-Turn Agent Trainingcited by 1
  22. 2026Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimizationcited by 1
  23. 2026Groupwise Agentic Grading and Advantage Redistribution for Code Agent RLcited by 1
  24. 2026Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purificationcited by 0
  25. 2026DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocodecited by 0
  26. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewardscited by 0
  27. 2026LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Promptscited by 0
  28. 2026Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVRcited by 0
  29. 2026HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segmentscited by 0
  30. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0
  31. 2026Improving the Diversity of LLM Outputs without a Trade-offcited by 0
  32. 2026Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RLcited by 0
  33. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0
  34. 2026Teach to Learn: Hint Annealing for Self-improving LLM Reasoningcited by 0
  35. 2026Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Modelscited by 0
  36. 2026Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVRcited by 0
  37. 2026Understanding Enrichment in Reinforcement Learningcited by 0
  38. 2026A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026P^2O: Joint Policy and Prompt Optimization
  2. 2026Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairing
  3. 2026LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
  4. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning
  5. 2025Zero Reinforcement Learning Towards General Domains
  6. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  7. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
  8. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts