Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner
On the map: Deep reinforcement learning methods → RL training of language and agent models
Abstract (arXiv)
Constraints are essential for stabilizing reinforcement learning fine-tuning (RFT) and preventing degenerate outputs, yet they inherently conflict with the optimization objective because stronger constraints limit the ability of a fine-tuned model to discover better solutions. We propose textitdynamic constraints that resolve this tension by adapting to the evolving capabilities of the fine-tuned model based on the insight that constraints should only intervene when degenerate outputs occur. We implement this by using a reference model as an textitonline refiner that takes the response from the fine-tuned model and generates a minimally corrected version which preserves correct content verbatim while fixing errors. A supervised fine-tuning loss then trains the fine-tuned model to produce the refined output. This mechanism yields a constraint that automatically strengthens or relaxes based on output quality. Experiments on dialogue and code generation show that dynamic constraints outperform both KL regularization and unconstrained baselines, achieving substantially higher task rewards while maintaining training stability.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 30 references that are in the map. First posted on arXiv 18 Mar 2026.
Builds on
20 of 30 references.
References · 20
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2020Learning to summarize from human feedbackcited by 2,588
- 2015Trust Region Policy Optimizationcited by 2,580
- 2021Measuring Coding Challenge Competence With APPScited by 902
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
- 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
- 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
- 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
- 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
- 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
- 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
8 more
- 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
- 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
- 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 59
- 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
- 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
- 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 30
- 2023The Importance of Human-Labeled Data in the Era of LLMscited by 8
- 2024Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewardscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
- 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
- 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
- 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
- 2026Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
- 2025RL's Razor: Why Online Reinforcement Learning Forgets Less
- 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
- 2025Anchored Supervised Fine-Tuning