mlsanityData to 14 Oct 2026

Qsharp: Provably Optimal Distributional RL for LLM Post-Training

arXiv 2502.20548 ↗First posted 27 Feb 2025cs.LG cs.AI cs.CLcited by 15

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) post-training is crucial for LLM alignment and reasoning, but existing policy-based methods, such as PPO and DPO, can fall short of fixing shortcuts inherited from pre-training. In this work, we introduce Qsharp, a value-based algorithm for KL-regularized RL that guides the reference policy using the optimal regularized Q function. We propose to learn the optimal Q function using distributional RL on an aggregated online dataset. Unlike prior value-based baselines that guide the model using unregularized Q-values, our method is theoretically principled and provably learns the optimal policy for the KL-regularized RL problem. Empirically, Qsharp outperforms prior baselines in math reasoning benchmarks while maintaining a smaller KL divergence to the reference policy. Theoretically, we establish a reduction from KL-regularized RL to no-regret online learning, providing the first bounds for deterministic MDPs under only realizability. Thanks to distributional RL, our bounds are also variance-dependent and converge faster when the reference policy has small variance. In sum, our results highlight Qsharp as an effective approach for post-training LLMs, offering both improved performance and theoretical guarantees. The code can be found at https://github.com/jinpz/qsharp.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 71 of its 92 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

71 of 92 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Entropy-Regularized Process Reward Model15 Dec
  2. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  4. 2023Let's Verify Step by Step31 May
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 65

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2017Decoupled Weight Decay Regularizationcited by 1,831
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  11. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
38 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  4. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  5. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  6. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  7. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  8. 2024Iterative Reasoning Preference Optimizationcited by 246
  9. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 233
  10. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  11. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  12. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  13. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  14. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  15. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 178
  16. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  17. 2024The pitfalls of next-token predictioncited by 160
  18. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  19. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 136
  20. 2023Controlled Decoding from Language Modelscited by 135
  21. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  22. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  23. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  24. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  25. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 90
  26. 2024Tuning Language Models by Proxycited by 89
  27. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  28. 2017Failures of Gradient-Based Deep Learningcited by 80
  29. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  30. 2018On Oracle-Efficient PAC RL with Rich Observationscited by 71
  31. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 71
  32. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  33. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  34. 2021Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discriminationcited by 53
  35. 2023Learning to Generate Better Than Your LLMcited by 49
  36. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  37. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  38. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 43

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resourcescited by 39
  3. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 24
  4. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 13
  5. 2026Discrete Adjoint Matchingcited by 11
  6. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  7. 2025textttSPECS: Faster Test-Time Scaling through Speculative Draftscited by 3
  8. 2025Efficient Controllable Diffusion via Optimal Classifier Guidancecited by 2
  9. 2025PITA: Preference-Guided Inference-Time Alignment for LLM Post-Trainingcited by 2
  10. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  11. 2025Reasoning without Regretcited by 0
  12. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
3 more
  1. 2026DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Trainingcited by 0
  2. 2026A Regret Minimization Framework on Preference Learning in Large Language Modelscited by 0
  3. 2026What Pretraining and Midtraining Make Learnable from Rewards?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  2. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
  3. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
  4. 2024Reinforcement Learning for LLM Post-Training: A Survey
  5. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
  6. 2025LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment
  7. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
  8. 2026MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning