mlsanityData to 7 Oct 2026

Optimizing Language Models for Inference Time Objectives using Reinforcement Learning

arXiv 2503.19595 ↗First posted 25 Mar 2025cs.LGcited by 49

Not on the map. Why?

Cited by 26 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we investigate the merits of explicitly optimizing for inference time algorithmic performance during model training. We show how optimizing for inference time performance can improve overall model efficacy. We consider generic inference time objectives with k samples, with a focus on pass@k and majority voting as two main applications. With language model training on reasoning datasets, we showcase the performance trade-off enabled by training with such objectives. When training on code generation tasks, we show that the approach significantly improves pass@k objectives compared to the baseline method.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

32 of 50 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
20 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  6. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  7. 2015Importance Weighted Autoencoderscited by 454
  8. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  9. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  10. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  11. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  12. 2016Variational inference for Monte Carlo objectivescited by 83
  13. 2024Variational Best-of-N Alignmentcited by 49
  14. 2024InfAlign: Inference-aware language model alignmentcited by 34
  15. 2019Improving Policies via Search in Cooperative Partially Observable Gamescited by 33
  16. 2024What Makes Large Language Models Reason in (Multi-Turn) Code Generation?cited by 29
  17. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 25
  18. 2014Techniques for Learning Binary Stochastic Feedforward Neural Networkscited by 14
  19. 2021Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluationcited by 5
  20. 2021Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 30.

49 papers.

Sort:

By area:

  1. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  2. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 87
  3. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  4. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  5. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 78
  6. 2025Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGIcited by 34
  7. 2026Maximum Likelihood Reinforcement Learningcited by 29
  8. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 19
  9. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  10. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  11. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Themcited by 14
  12. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
37 more
  1. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 12
  2. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMscited by 12
  3. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 11
  4. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 9
  5. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 9
  6. 2025Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectoriescited by 8
  7. 2025Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scalingcited by 8
  8. 2026Thinking with Map: Reinforced Parallel Map-Augmented Agent for Geolocalizationcited by 8
  9. 2025Polychromic Objectives for Reinforcement Learningcited by 7
  10. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  11. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 7
  12. 2025Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoningcited by 6
  13. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 6
  14. 2025GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Timecited by 5
  15. 2026Beyond Mode Collapse: Distribution Matching for Diverse Reasoningcited by 5
  16. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  17. 2026Regret Analysis of Retry-Based Banditscited by 4
  18. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 3
  19. 2026SPIRAL: Learning to Search and Aggregatecited by 3
  20. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  21. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  22. 2026Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generationcited by 2
  23. 2026Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Requiredcited by 2
  24. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPOcited by 1
  25. 2026UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMscited by 1
  26. 2026Retry Policy Gradients in Continuous Action Spacescited by 1
  27. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 1
  28. 2026What should post-training optimize? A test-time scaling law perspectivecited by 0
  29. 2026REVES: REvision and VErification--Augmented Training for Test-Time Scalingcited by 0
  30. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  31. 2026DecompRL: Solving Harder Problems by Learning Modular Code Generationcited by 0
  32. 2026Theoretical Foundations of max@k Reinforcement Learningcited by 0
  33. 2026Direct Optimization of Generators for Search in Automated Theorem Provingcited by 0
  34. 2026Planned Test-Time Scaling with Coordinated Reasoning Pathscited by 0
  35. 2026The Statistical Benefits of Multiple Responses for Learning from Demonstrationscited by 0
  36. 2026Composable Decoding on the Probability Simplex: Theory and Implementationcited by 0
  37. 2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Exploration-Driven Optimization for Test-Time Large Language Model Reasoning
  2. 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
  3. 2025Training Language Models to Reason Efficiently
  4. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  5. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  6. 2026Compute Aligned Training: Optimizing for Test Time Inference
  7. 2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals
  8. 2025TTRL: Test-Time Reinforcement Learning