mlsanityData to 7 Oct 2026

Supervised Optimism Correction: Be Confident When LLMs Are Sure

arXiv 2504.07527 ↗First posted 10 Apr 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

In this work, we establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning under the token-level Markov decision process, revealing that large language models indeed learn an implicit Q-function for inference. Through this theoretical lens, we demonstrate that the widely used beam search method suffers from unacceptable over-optimism, where inference errors are inevitably amplified due to inflated Q-value estimations of suboptimal steps. To address this limitation, we propose Supervised Optimism Correction(SOC), which introduces a simple yet effective auxiliary loss for token-level Q-value estimations during supervised fine-tuning. Specifically, the auxiliary loss employs implicit value regularization to boost model confidence in expert-demonstrated responses, thereby suppressing over-optimism toward insufficiently supervised responses. Extensive experiments on mathematical reasoning benchmarks, including GSM8K, MATH, and GAOKAO, showcase the superiority of the proposed SOC with beam search across a series of open-source models.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

43 of 61 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
31 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  9. 2024A Survey on Large Language Models for Code Generationcited by 646
  10. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  11. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  12. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  13. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  14. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  15. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  16. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  17. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  18. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  19. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  20. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  21. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 88
  22. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  23. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
  24. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  25. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  26. 2025A Survey of Direct Preference Optimizationcited by 35
  27. 2025Reasoning with Reinforced Functional Token Tuningcited by 14
  28. 2020Directed Beam Search: Plug-and-Play Lexically Constrained Language Generationcited by 12
  29. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  30. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
  31. 2023Allies: Prompting Large Language Model with Beam Searchcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13
  2. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Modelscited by 7
  3. 2025Rethinking Supervised Fine-Tuning: Emphasizing Key Answer Tokens for Improved LLM Accuracycited by 4
  4. 2026STRIDE: Learnable Stepwise Language Feedback for LLM Reasoningcited by 1
  5. 2026How Does Prefix Matter in Reasoning Model Tuning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
  2. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
  3. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuning
  4. 2025General Exploratory Bonus for Optimistic Exploration in RLHF
  5. 2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning
  6. 2026Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs
  7. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
  8. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning