Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent
Not on the map. Why?
Abstract (arXiv)
Online reinforcement learning with verifiable rewards (RLVR) turns checkable outcomes into a scalable training signal, but it keeps rollout generation, verifier scoring, and reference-policy evaluations on the optimization path. Static weighted supervised fine-tuning (SFT) on precomputed rollouts seems to remove this bottleneck, yet a weighted likelihood is not specified by rewards alone: its sampler and weights induce the policy being fit. This paper identifies the reference-sampled weighted-SFT objective whose induced policy equals the fixed-reference KL-regularized RLVR optimizer. The optimizer is the standard Boltzmann target policy, obtained by exponentially tilting the reference policy by verifier reward. Matching a weighted-SFT induced policy to this target forces density-ratio weights; in the reference-sampled subclass, this reduces uniquely, up to prompt scaling, to the prompt-normalized Boltzmann weight exp(r(x,y)/β)/Z(x). BOLT, a Boltzmann-Targeted SFT procedure, is the empirical estimator of this projection. The finite one-shot analysis separates the exact stored-support price βlog(1/π^*(SNmid x)) from partition estimation, effective-sample-size variance, generalization, optimization, and approximation errors. This decomposition explains why extra SFT epochs cannot repair missing reference-policy coverage and exposes the temperature--coverage--variance frontier. When coverage needs adaptive sampling, refreshed Boltzmann projections become KL policy mirror descent; finite inner solves enter as additive drift from the exact mirror step. Single-run Qwen experiments provide projection evidence for the target-matched weight, one-shot saturation, refreshed-sampler gains, and optimization-time savings, within the stated single-run scope.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 4 May 2026.
Builds on
45 of 59 references.
References · 45
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
33 more
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2020Learning to summarize from human feedbackcited by 2,562
- 2015Trust Region Policy Optimizationcited by 2,503
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
- 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
- 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
- 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
- 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
- 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
- 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
- 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
- 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
- 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
- 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
- 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 281
- 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
- 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
- 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
- 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
- 2025Agent Learning via Early Experiencecited by 79
- 2023Empirical Design in Reinforcement Learningcited by 54
- 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
- 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)cited by 37
- 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
- 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10
Built on by
Citing papers per year: 2026: 1.
1 paper.