mlsanityData to 14 Oct 2026

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

arXiv 2608.25756 ↗First posted 26 Aug 2026cs.LG cs.AIcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT often improves pass@16 performance on math and coding evaluations, with gains up to 17% absolute, while incurring minimal computational overhead. These higher-coverage checkpoints consistently translate to up to 4% absolute pass@1 gains in subsequent GRPO runs, demonstrating that TailSFT checkpoints serve as better initializations for RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 55 of its 70 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

55 of 70 references.

References · 55

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  5. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  6. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  7. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  9. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  10. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  11. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  12. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
38 more
  1. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  2. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  3. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  4. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  5. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 129
  6. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  7. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  8. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  9. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  10. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  11. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  12. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  13. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 89
  14. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  15. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
  16. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  17. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 80
  18. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 71
  19. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  20. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 69
  21. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 62
  22. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 60
  23. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  24. 2025Overtrained Language Models Are Harder to Fine-Tunecited by 58
  25. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 41
  26. 2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
  27. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)cited by 38
  28. 2025Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Explorationcited by 36
  29. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  30. 2025The Coverage Principle: How Pre-Training Enables Post-Trainingcited by 30
  31. 2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoningcited by 29
  32. 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 29
  33. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 28
  34. 2024CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-trainingcited by 20
  35. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  36. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 13
  37. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  38. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 10

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026RL Starts before RL: On Policy Distillation for Better Reinforcement Learningcited by 3
  2. 2026From Imitation to Reward Discovery: On-Policy Warmup for Agentic RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Tail-Likelihood Reinforcement Learning
  2. 2025Anchored Supervised Fine-Tuning
  3. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
  4. 2026PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost
  5. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  6. 2026Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback
  7. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
  8. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy