mlsanityData to 7 Oct 2026

ROSS: Relearning from Self-Generated Rollouts through Selective Supervision

arXiv 2609.35954 ↗First posted 28 Sep 2026cs.LG cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-play RL for reasoning also fits: Self-rewarding alignment and self-training

Abstract (arXiv)

Large language model post-training generates self-generated rollouts through reinforcement learning and on-policy distillation, yet this experience is often treated as stale once the policy advances. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer-grained selective supervision. We introduce ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations. Across domain-specific reinforcement learning, multi-teacher on-policy distillation, and agentic reinforcement learning, ROSS consistently improves upstream checkpoints and outperforms baselines across mathematics, code generation, instruction following, and software engineering. On Qwen3.6-35B-A3B, ROSS improves the six-benchmark MOPD average from 58.40% to 62.20% and SWE-bench Verified from 64.20% to 68.40%. These results show that self-rollout training leaves behind reusable behavioral experience that can yield further gains through offline supervised fine-tuning (SFT), without additional policy rollouts.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-play RL for reasoning, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 28 of its 34 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

28 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun

Other references · 27

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  9. 2025Group Sequence Policy Optimizationcited by 788
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  11. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
15 more
  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  3. 2025Process Reward Models That Thinkcited by 118
  4. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  5. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  6. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 64
  7. 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Predictioncited by 54
  8. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 53
  9. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 26
  10. 2025Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Maskingcited by 16
  11. 2026daVinci-Env: Open SWE Environment Synthesis at Scalecited by 13
  12. 2026RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuningcited by 3
  13. 2026SWE-Prime: Fewer Trajectories, Better Performancecited by 3
  14. 2025RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initializationcited by 2
  15. 2026Step Rejection Fine-Tuning: A Practical Distillation Recipecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  2. 2026SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
  3. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
  4. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions
  5. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
  6. 2026DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training
  7. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
  8. 2026Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models