mlsanityData to 7 Oct 2026

CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution

arXiv 2603.17775 ↗First posted 18 Mar 2026cs.CL cs.AI cs.LGcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 51 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

47 of 51 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization2 Oct
  2. 2025A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning18 Jul
  3. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models5 Jun
  4. 2025Maximizing Confidence Alone Improves Reasoning28 May
  5. 2025Learning to Reason without External Rewards26 May
  6. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  7. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  8. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning18 Feb
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  12. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
2 more
  1. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  2. 2023Let's Verify Step by Step31 May

Other references · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2024OpenAI o1 System Cardcited by 1,220
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
21 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2024Self-Rewarding Language Modelscited by 654
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  4. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  5. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  6. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  7. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  8. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  9. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  10. 2025Self-rewarding correction for mathematical reasoningcited by 67
  11. 2024Self-Consistency Preference Optimizationcited by 50
  12. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  13. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewardscited by 38
  14. 2025Toward Training Superintelligent Software Agents through Self-Play SWE-RLcited by 37
  15. 2025ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verificationcited by 35
  16. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  17. 2025GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulatorscited by 34
  18. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  19. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 27
  20. 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistencycited by 20
  21. 2025Too Consistent to Detect: A Study of Self-Consistent Errors in LLMscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Milestone-Guided Policy Learning for Long-Horizon Language Agentscited by 12
  2. 2026Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Votingcited by 5
  3. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  4. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1
  5. 2026Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Modelscited by 1
  6. 2026Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learningcited by 1
  7. 2026Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Designcited by 1
  8. 2026Agent-G2: Gaussian Guidance for Agentic Reinforcement Learningcited by 0
  9. 2026Rewarding Reasoning, Not Answers: Fixing and Bounding Test-Time Reinforcement Learning on Medical QAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Consensus as Privileged Context for Label-Free Self-Distillation
  2. 2026Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism
  3. 2025You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models
  4. 2025VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains
  5. 2026SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology
  6. 2025Socratic-Zero : Bootstrapping Reasoning via Data-Free Agent Co-evolution
  7. 2026MetaCtrl: Your Large Language Models Can Reason Better and More Concisely with a Metacognitive Controller
  8. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time