mlsanityData to 7 Oct 2026

Verifier-Induced Support Reshaping in On-Policy Optimization

arXiv 2608.00220 ↗First posted 31 Jul 2026cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce. We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget. Across two model families, we study this effect through repeated verifier-scored sampling and bidirectional training on mathematical reasoning and constrained instruction following, including sequential training with the opposite verifier. Math-RLVR raises average instruction-following success but reduces the number of prompts with any successful response under repeated sampling. On IFEval with Qwen3-8B-Base, pass@1 rises by 6.5 percentage points while best@32 falls by 9.8 percentage points, and the same divergence appears across both models and IF benchmarks. Conversely, IF-RLVR shifts math responses from step-by-step openings toward direct answers, lowers best@k across sampling budgets, and reduces reward variation for later Math-RLVR. Token-distribution analyses and controlled opening interventions show that these changes concentrate in the first few response tokens. RLVR mainly reranks openings already available in the base policy, and the selected opening causally affects math searchability. The tested reference-policy constraints, routing priors, and on-policy distillation preserve cross-task support only partially; MathIF and ReasonIF show that marginal gains translate only partly into responses that are both correct and constraint-following. Therefore, endpoint improvements do not guarantee future trainability or joint capability under on-policy optimization. Code is available at https://github.com/sylvain-wei/VISR

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 35 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

32 of 35 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
20 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  3. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  4. 2025Generalizing Verifiable Instruction Followingcited by 178
  5. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  6. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  7. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  8. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  9. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 112
  10. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  11. 2025Overcoming catastrophic forgetting in neural networkscited by 52
  12. 2025When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMscited by 51
  13. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 43
  14. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
  15. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  16. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMscited by 20
  17. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  18. 2025TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarioscited by 14
  19. 2025ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoningcited by 12
  20. 2025IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewardscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026On-Policy Visual Evidence Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
  2. 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
  3. 2026From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL
  4. 2026On-policy Distillation with Verifiable Reward
  5. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
  6. 2025From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning
  7. 2026Reinforcement Learning via Self-Distillation
  8. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers