mlsanityData to 7 Oct 2026

Maximum Likelihood Reinforcement Learning

arXiv 2602.02710 ↗First posted 2 Feb 2026cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) is the method of choice for training models in setups where the objective function can only be evaluated by sampling from the model. Our key observation is that when the feedback is terminal and binary, models implicitly induce a likelihood over correct rollouts. Maximum likelihood would be the natural framework in such settings, but RL is used instead as a workaround to the non-differentiability. We prove that the standard, expected-reward RL formulation is only a first-order approximation of the likelihood. To remedy this mismatch, we introduce Maximum Likelihood Reinforcement Learning (MaxRL), a compute-indexed family of sample-based objectives that interpolate between expected-reward RL and maximum likelihood as sampling compute is scaled. The resulting objective is a one-line change to standard RL implementations. MaxRL Pareto-dominates existing methods in all tested models and tasks, achieves up to 20× gains in test-time scaling efficiency over GRPO, and scales more favorably with additional training data and compute.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 124 of its 200 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

124 of 200 references.

References · 124

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2016Generative Adversarial Imitation Learningcited by 1,576
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  13. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  14. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  15. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  16. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  17. 2016End to End Learning for Self-Driving Carscited by 1,000
  18. 2018Exploration by Random Network Distillationcited by 966
  19. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  20. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  21. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  22. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  23. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  24. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  25. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  26. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  27. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  28. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  29. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  30. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  31. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  32. 2020Reinforcement Learning with Augmented Datacited by 452
  33. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  34. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  35. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  36. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  37. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  38. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322

50 most cited of 124 All 124 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 29.

29 papers.

Sort:

By area:

  1. 2026Composer 2 Technical Reportcited by 20
  2. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 9
  3. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  4. 2026Not All LLM Reasoning is Visible in the Chain-of-Thoughtcited by 5
  5. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 3
  6. 2026SPIRAL: Learning to Search and Aggregatecited by 3
  7. 2026Policy Improvement Reinforcement Learningcited by 2
  8. 2026Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplexcited by 2
  9. 2026Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirementscited by 2
  10. 2026F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Rankingcited by 1
  11. 2026Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewardscited by 1
  12. 2026Spectral Rewiring for Exploration, Purification, and Model Mergingcited by 1
17 more
  1. 2026Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Mergingcited by 1
  2. 2026SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RLcited by 1
  3. 2026Learning to Solve Hard Problems in RL for LLMs by Never Giving Upcited by 1
  4. 2026Target Policy Optimizationcited by 0
  5. 2026How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuumcited by 0
  6. 2026CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoningcited by 0
  7. 2026RL2ML: Finite-Rollout Surrogate Objectives from Reinforcement Learning to Maximum Likelihoodcited by 0
  8. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  9. 2026Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignmentcited by 0
  10. 2026ISO: An RLVR-Native Optimization Stackcited by 0
  11. 2026Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoningcited by 0
  12. 2026SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimationcited by 0
  13. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0
  14. 2026LookThere! Sparse Vision by Reinforced Selectioncited by 0
  15. 2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0
  16. 2026Measuring and Mitigating Solution Mode Collapse in RLVRcited by 0
  17. 2026Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds
  2. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
  3. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  4. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  5. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design
  6. 2023The In-Sample Softmax for Offline Reinforcement Learning
  7. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  8. 2026Reward-Conditioned Reinforcement Learning