mlsanityData to 14 Oct 2026

Strategically Diverse Sampling for Self-Training

arXiv 2609.31571 ↗First posted 25 Sep 2026cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning also fits: Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Many LLM training and inference methods, including RL and test-time scaling, depend on repeated sampling, but benefit only when the responses meaningfully differ. Self-training faces the same challenge: training data is typically constructed by sampling IID responses and filtering primarily for correctness, thereby overrepresenting strategies a model already favours. We investigate strategic diversity, or substantive variation among approaches to a problem, as an alternative principle for constructing self-training data. We generate strategically diverse data with two sampling methods: GROOT, a new method which constructs a hierarchical tree of approaches and samples distinct paths, and Verbalized Sampling (VS), adapted to produce an unstructured set of approaches. Across competitive programming and Next-Chapter Prediction domains, models trained on strategically sampled data outperform IID-trained counterparts on difficult tasks and provide strong initializations for RL and test-time scaling. Most strikingly, self-training on strategically diverse but incorrect traces from Qwen3-4B outperforms IID distillation from a 235B teacher. These results challenge prevailing assumptions about what makes useful self-training data and show that diversity of approaches can matter more than correctness or teacher scale.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 50 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

46 of 50 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Hint-Guided Diversified Policy Optimization for LLM Reasoning2 Jun
  2. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning1 Feb
  3. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning25 Nov
  4. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  5. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy16 Jun
  6. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  9. 2025Evaluating the Diversity and Quality of LLM Generated Content16 Apr

Other references · 37

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  8. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  9. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  10. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  11. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  12. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
25 more
  1. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
  2. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 105
  3. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  4. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  5. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 65
  6. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  7. 2025Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Modelscited by 35
  8. 2025Learning to Reason for Long-Form Story Generationcited by 34
  9. 2025OJBench: A Competition Level Code Benchmark For Large Language Modelscited by 26
  10. 2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
  11. 2025On the Effect of Sampling Diversity in Scaling LLM Inferencecited by 18
  12. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  13. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  14. 2026Where does output diversity collapse in post-training?cited by 11
  15. 2025Task-Dependent Evaluation of LLM Output Homogenization: A Taxonomy-Guided Frameworkcited by 9
  16. 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversitycited by 9
  17. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  18. 2026Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoderscited by 6
  19. 2026DIVE: Unlocking Self-Improvement in Frozen Language Models Through Diversity-Driven Skill Evolutioncited by 5
  20. 2026Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generationcited by 4
  21. 2026Diversity in Large Language Models under Supervised Fine-Tuningcited by 4
  22. 2026Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectivescited by 3
  23. 2026Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoningcited by 2
  24. 2026Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Modelscited by 1
  25. 2026Learning to Solve Hard Problems in RL for LLMs by Never Giving Upcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning
  2. 2026The Two-Stage Decision-Sampling Hypothesis: Understanding the Emergence of Self-Reflection in RL-Trained LLMs
  3. 2026R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training
  4. 2026Test-Time Scaling Makes Overtraining Compute-Optimal
  5. 2025A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning
  6. 2025Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs
  7. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute
  8. 2026Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling