mlsanityData to 7 Oct 2026

Tandem Reinforcement Learning with Verifiable Rewards

arXiv 2606.28166 ↗First posted 26 Jun 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has driven substantial gains in language-model reasoning. However, models fine-tuned with RLVR need not remain compatible with weaker partners: gains achieved alone may erode under collaboration, while policy drift may place reasoning beyond a weaker model's predictive reach. To address both problems, we introduce Tandem RLVR (TRLVR), bringing the recently proposed tandem-training paradigm beyond proof-of-concept settings and into modern RLVR. In TRLVR, a senior learns from rewarded trajectories co-generated with a frozen junior, while the standard GRPO objective is applied only to senior-emitted tokens. Training Qwen3-4B-Instruct on competition math, TRLVR performs on par with a matched GRPO senior when reasoning alone. We characterize compatibility through the complementary lenses of handoff robustness and junior legibility. Under reasoning-step handoffs, the TRLVR senior retains nearly all of its solo performance with the junior, reducing the communication tax to a negligible loss. The senior's reasoning is also more legible to the junior: its token distribution stays more closely anchored to the junior's, and the junior predicts the senior's chain-of-thought more readily token by token. An ablation that explicitly regularizes GRPO toward the junior does not recover the same combination of capability and compatibility. These results provide initial evidence that tandem rollout structure is a promising direction for keeping RLVR reasoning gains accessible to weaker partners.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 62 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

53 of 62 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning8 Apr
  2. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs23 Mar
  3. 2025Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards5 Nov
  4. 2025ExGRPO: Learning to Reason from Experience2 Oct
  5. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse30 Sep
  6. 2025Geometric-Mean Policy Optimization28 Jul
  7. 2025Reinforcement Learning Teachers of Test Time Scaling10 Jun
  8. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  9. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2023Let's Verify Step by Step31 May
2 more
  1. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  2. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  11. 2025Group Sequence Policy Optimizationcited by 788
  12. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
27 more
  1. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  2. 2022Teaching Small Language Models to Reasoncited by 279
  3. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  4. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  5. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  6. 2020Aligning Superhuman AI with Human Behavior: Chess as a Model Systemcited by 78
  7. 2025Latent Collaboration in Multi-Agent Systemscited by 67
  8. 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 63
  9. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  10. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 30
  11. 2023Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Noveltycited by 27
  12. 2024Maia-2: A Unified Model for Human-AI Alignment in Chesscited by 24
  13. 2025MARS: toward more efficient multi-agent collaboration for LLM reasoningcited by 13
  14. 2025The Impact of Language Mixing on Bilingual LLM Reasoningcited by 11
  15. 2025The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperationcited by 11
  16. 2025Output Supervision Can Obfuscate the Chain of Thoughtcited by 11
  17. 2023SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classificationcited by 10
  18. 2024Designing Skill-Compatible AI: Methodologies and Frameworks in Chesscited by 10
  19. 2025The Steganographic Potentials of Language Modelscited by 10
  20. 2026LLM Safety From Within: Detecting Harmful Content with Internal Representationscited by 10
  21. 2025Reasoning Models Sometimes Output Illegible Chains of Thoughtcited by 9
  22. 2025Learning to Imitate with Less: Efficient Individual Behavior Modeling in Chesscited by 7
  23. 2026Analyzing and Improving Chain-of-Thought Monitorability Through Information Theorycited by 7
  24. 2026Measuring Weak-to-Strong Legibility of Reasoning Modelscited by 5
  25. 2026ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinementcited by 5
  26. 2025SSR: Speculative Parallel Scaling Reasoning in Test-timecited by 4
  27. 2025Tandem Training for Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
  2. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  4. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  5. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  6. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  7. 2026Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewards
  8. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers