mlsanityData to 14 Oct 2026

Optimal Policy Minimum Bayesian Risk

arXiv 2505.17242 ↗First posted 22 May 2025cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Inference scaling helps LLMs solve complex reasoning problems through extended runtime computation. On top of long chain-of-thought (long-CoT) models, purely inference-time techniques such as best-of-N (BoN) sampling, majority voting, or more generally, minimum Bayes risk decoding (MBRD), can further improve LLM accuracy by generating multiple candidate solutions and aggregating over them. These methods typically leverage additional signals in the form of reward models and risk/similarity functions that compare generated samples, e.g., exact match in some normalized space or standard similarity metrics such as Rouge. Here we present a novel method for incorporating reward and risk/similarity signals into MBRD. Based on the concept of optimal policy in KL-controlled reinforcement learning, our framework provides a simple and well-defined mechanism for leveraging such signals, offering several advantages over traditional inference-time methods: higher robustness, improved accuracy, and well-understood asymptotic behavior. In addition, it allows for the development of a sample-efficient variant of MBRD that can adjust the number of samples to generate according to the difficulty of the problem, without relying on majority vote counts. We empirically demonstrate the advantages of our approach on math (MATH-500) and coding (HumanEval) tasks using recent open-source models. We also present a comprehensive analysis of its accuracy-compute trade-offs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 43 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

31 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  3. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  4. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  5. 2023Let's Verify Step by Step31 May
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  9. 2024Phi-4 Technical Reportcited by 711
  10. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  12. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
11 more
  1. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  2. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  3. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  4. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 115
  5. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 82
  6. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  7. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 70
  8. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 51
  9. 2023Faster Minimum Bayes Risk Decoding with Confidence-based Pruningcited by 34
  10. 2024BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedbackcited by 7
  11. 2024Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translationcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Latency and Token-Aware Test-Time Computecited by 8
  2. 2025Confidence-Weighted Token Set Cover for Early Hypothesis Pruning in Self-Consistencycited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning
  2. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  3. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
  4. 2026Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning
  5. 2025Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scaling
  6. 2026Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
  7. 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning
  8. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning