mlsanityData to 14 Oct 2026

Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs

arXiv 2609.01573 ↗First posted 1 Sep 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open problem. Existing work characterizes only broad trends (e.g., SFT dominates in low-data regimes), lacks a principled allocation framework, and does not examine whether the optimal ratio transfers across model sizes. We frame this problem in terms of near-optimality: rather than seeking a single optimal SFT-RL ratio, we characterize the near-optimal region, the set of allocations within a specified tolerance of peak performance. Empirically, this region is wide even for small tolerances (2-10%), widens with model scale, and transfers reliably from small proxy models to large target models. This yields a practical strategy: small proxy-model experiments suffice to identify a transferable near-optimal region, eliminating the need for exhaustive large-scale search. Our results hold consistently across tasks, model families, and both preference-based off-policy and reward-supervision on-policy RL methods. We further analyze how the asymmetry in annotation costs between SFT and RL data shifts the near-optimal region.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

47 of 59 references.

References · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2023LIMA: Less Is More for Alignmentcited by 1,100
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
35 more
  1. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  2. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  4. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  5. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  6. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  7. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  8. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  9. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  10. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  11. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  12. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  13. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  14. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 90
  15. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  16. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  17. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
  18. 2024Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?cited by 40
  19. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 30
  20. 2021Pre-train or Annotate? Domain Adaptation with a Constrained Budgetcited by 27
  21. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  22. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
  23. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  24. 2025DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoningcited by 15
  25. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
  26. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  27. 2025Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Frameworkcited by 9
  28. 2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8
  29. 2025AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMscited by 8
  30. 2025ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimizationcited by 5
  31. 2025Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuningcited by 4
  32. 2025AutoMixer: Checkpoint Artifacts as Automatic Data Mixerscited by 4
  33. 2025ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selectioncited by 3
  34. 2025DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agentscited by 2
  35. 2026Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversionscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
  2. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
  3. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
  4. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  5. 2026IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL
  6. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs
  7. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
  8. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)