mlsanityData to 7 Oct 2026

Selecting Diverse SFT Traces Improves Post-RL Generalization

arXiv 2609.33780 ↗First posted 27 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Verified solutions are not equally useful for preparing reasoning models for reinforcement learning (RL). We present a comprehensive study of route diversity, the variation in the sequences of reasoning steps in supervised fine-tuning (SFT) data, and propose a lightweight, rule-based fingerprint to select for it. From one pool at one budget, with matched training recipes and checkpoints, selecting diverse rather than similar routes improves post-RL problem coverage across puzzles and mathematics, including on problems harder than those seen in either training stage. In synthetic experiments, route-diverse SFT improves OLMo3-7B's pass@8 by 16.9 points on environments held out from SFT. In a single-model condition, where one model writes every candidate, diverse selection gains up to 6.2 points of mean pass@8 across 10 mathematics benchmarks. Pre-RL diagnostics suggest why: diverse SFT can produce both successful and failed attempts on more prompts despite slightly lower mean accuracy, giving group-relative RL more prompts with a learning signal. On 3 open-source corpora, our CPU-only selector, without model calls, outperforms more expensive alternatives in every comparison of mean post-RL performance. These results identify reasoning-route diversity as a practical criterion for selecting SFT data that better prepares models for RL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 92 of its 112 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

92 of 112 references.

Closest ancestors · 30

Same direction, newest first

  1. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model26 Mar
  2. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models2 Feb
  3. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning1 Feb
  4. 2026Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs13 Jan
  5. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models8 Dec
  6. 2025VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL24 Nov
  7. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learning16 Nov
  8. 2025The Path Not Taken: RLVR Provably Learns Off the Principals11 Nov
  9. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead2 Oct
  10. 2025Diversity-Incentivized Exploration for Versatile Reasoning30 Sep
  11. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  12. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models24 Sep
18 more
  1. 2025RL Fine-Tuning Heals OOD Forgetting in SFT8 Sep
  2. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  3. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  4. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy16 Jun
  5. 2025Magistral12 Jun
  6. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning7 Jun
  7. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  9. 2025Behavior Injection: Preparing Language Models for Reinforcement Learning25 May
  10. 2025Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning20 May
  11. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining12 May
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  13. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  14. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  15. 2025Process Reinforcement through Implicit Rewards3 Feb
  16. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  17. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  18. 2023Let's Verify Step by Step31 May

Other references · 62

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2025s1: Simple test-time scalingcited by 1,311
  11. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
38 more
  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  5. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  8. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  9. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  10. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  11. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  12. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  13. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  14. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  15. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  16. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  17. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  18. 2025Generalizing Verifiable Instruction Followingcited by 178
  19. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  20. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  21. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  22. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 151
  23. 2025Phi-4-reasoning Technical Reportcited by 146
  24. 2024Stream of Search (SoS): Learning to Search in Languagecited by 141
  25. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  26. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  27. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 108
  28. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  29. 2024Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimationcited by 53
  30. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 53
  31. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 53
  32. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  33. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 48
  34. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  35. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 43
  36. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 38
  37. 2025NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Taskscited by 29
  38. 2025Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Propertiescited by 28

50 most cited of 62 All 62 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  2. 2026Diverse Thinking Schemata Elicit Better Reasoning in Large Language Models
  3. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
  4. 2026Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models
  5. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning
  6. 2025Debunk the Myth of SFT Generalization
  7. 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity
  8. 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning