# Maximum Likelihood Reinforcement Learning

arXiv: https://arxiv.org/abs/2602.02710 · HTML: https://mlsanity.com/paper/2602.02710/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 124 of its 200 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 124)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)

## Built on by (most cited first, 30 of 36)

- [Composer 2 Technical Report](https://mlsanity.com/paper/2603.24477/) (2026)
- [Reinforcement Learning from Rich Feedback with Distributional DAgger](https://mlsanity.com/paper/2606.05152/) (2026)
- [Vector Policy Optimization: Training for Diversity Improves Test-Time Search](https://mlsanity.com/paper/2605.22817/) (2026)
- [Poly-EPO: Training Exploratory Reasoning Models](https://mlsanity.com/paper/2604.17654/) (2026)
- [Not All LLM Reasoning is Visible in the Chain-of-Thought](https://mlsanity.com/paper/2607.22925/) (2026)
- [High-Dimensional Statistics: Reflections on Progress and Open Problems](https://mlsanity.com/paper/2605.05076/) (2026)
- [Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex](https://mlsanity.com/paper/2605.06139/) (2026)
- [SPIRAL: Learning to Search and Aggregate](https://mlsanity.com/paper/2606.23595/) (2026)
- [Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements](https://mlsanity.com/paper/2608.17310/) (2026)
- [Policy Improvement Reinforcement Learning](https://mlsanity.com/paper/2604.00860/) (2026)
- [Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment](https://mlsanity.com/paper/2607.04728/) (2026)
- [ISO: An RLVR-Native Optimization Stack](https://mlsanity.com/paper/2607.19331/) (2026)
- [SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL](https://mlsanity.com/paper/2608.24870/) (2026)
- [Target Policy Optimization](https://mlsanity.com/paper/2604.06159/) (2026)
- [F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking](https://mlsanity.com/paper/2605.12995/) (2026)
- [Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2605.20865/) (2026)
- [RL2ML: Finite-Rollout Surrogate Objectives from Reinforcement Learning to Maximum Likelihood](https://mlsanity.com/paper/2605.30154/) (2026)
- [Spectral Rewiring for Exploration, Purification, and Model Merging](https://mlsanity.com/paper/2607.03065/) (2026)
- [Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning](https://mlsanity.com/paper/2608.02149/) (2026)
- [Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging](https://mlsanity.com/paper/2608.05541/) (2026)
- [SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation](https://mlsanity.com/paper/2608.09271/) (2026)
- [Learning to Solve Hard Problems in RL for LLMs by Never Giving Up](https://mlsanity.com/paper/2609.13443/) (2026)
- [How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum](https://mlsanity.com/paper/2604.25907/) (2026)
- [CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoning](https://mlsanity.com/paper/2605.24331/) (2026)
- [Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL](https://mlsanity.com/paper/2607.01490/) (2026)
- [Performance Foundations of Parallel & Distributed Reasoning Language Models](https://mlsanity.com/paper/2608.27046/) (2026)
- [LookThere! Sparse Vision by Reinforced Selection](https://mlsanity.com/paper/2609.04698/) (2026)
- [SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning](https://mlsanity.com/paper/2609.36552/) (2026)
- [Sharpening Tax in Post-Training](https://mlsanity.com/paper/2610.01509/) (2026)
- [Tropical Reinforcement Learning](https://mlsanity.com/paper/2610.02478/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) is the method of choice for training models in setups where the objective function can only be evaluated by sampling from the model. Our key observation is that when the feedback is terminal and binary, models implicitly induce a likelihood over correct rollouts. Maximum likelihood would be the natural framework in such settings, but RL is used instead as a workaround to the non-differentiability. We prove that the standard, expected-reward RL formulation is only a first-order approximation of the likelihood. To remedy this mismatch, we introduce Maximum Likelihood Reinforcement Learning (MaxRL), a compute-indexed family of sample-based objectives that interpolate between expected-reward RL and maximum likelihood as sampling compute is scaled. The resulting objective is a one-line change to standard RL implementations. MaxRL Pareto-dominates existing methods in all tested models and tasks, achieves up to 20× gains in test-time scaling efficiency over GRPO, and scales more favorably with additional training data and compute.