mlsanityData to 13 Oct 2026

Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR

arXiv 2609.37868 ↗First posted 29 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) methods such as GRPO rely on successful self-generated trajectories, but finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal. While additional rollouts improve the chance of success at higher cost, successful trajectories missing from one model's rollouts may already have been discovered by another. Indeed, we observe that heterogeneous models often succeed on complementary prompts, creating opportunities for mutual learning without a designated stronger teacher. To exploit this complementarity, we propose GRAFT (Gated Replacement of Answer-Failed groups with peer Trajectories), an off-policy-aware framework that replaces all-fail groups with informative peer groups. GRAFT transfers both successful and unsuccessful peer responses with peer-computed advantages, while controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance ratio clipping. Across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT consistently improves both models over GRPO with the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points in model-level average performance. Stored peer trajectories preserve most of the gains, improving over GRPO by 1.8 points on average without simultaneous co-training.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 32 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

29 of 32 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning31 May
  2. 2026F-TIS: Harnessing Diverse Models in Collaborative GRPO21 May
  3. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation20 May
  4. 2026Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards19 Mar
  5. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model21 Oct
  6. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs5 Oct
  7. 2025Rethinking Entropy Regularization in Large Reasoning Models29 Sep
  8. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  9. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization31 Jul
  10. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation17 Jul
  11. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
  12. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
8 more
  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  2. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025Learning to Reason under Off-Policy Guidance21 Apr
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov

Other references · 9

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  2. 2025Qwen3 Technical Reportcited by 8,510
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,369
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 873
  6. 2025Group Sequence Policy Optimizationcited by 806
  7. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 254
  8. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 14
  9. 2026Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance
  2. 2026OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning
  3. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  4. 2026Reinforcement Learning with Decomposed Subtasks
  5. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  6. 2026From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL
  7. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  8. 2026When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion