mlsanityData to 7 Oct 2026

GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

arXiv 2605.26184 ↗First posted 25 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Hybrid post-training usually combines supervised fine-tuning and reinforcement learning, but fixed mixing schedules cannot adapt when the relative noise of the two signals changes over time. We propose GAC, a noise-aware controller that derives an adaptive mixing weight from online estimates of gradient variance and disagreement between the two training signals. The method adds smoothing, prior guidance, and bounded updates while reusing existing training tensors. Experiments on math, code, science, and logic benchmarks show that GAC consistently improves hybrid post-training over strong fixed and rule-based baselines, with larger gains at larger model scales and less than 1% training overhead.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

27 of 35 references.

References · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
15 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  2. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  3. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  6. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  7. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  8. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  9. 2025Learning to Reason under Off-Policy Guidancecited by 266
  10. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  11. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  12. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 80
  13. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  14. 2023Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithmscited by 33
  15. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-trainingcited by 6

Built on by

Not cited yet.