# An Expectation-Maximization Perspective on Reinforcement Learning for LLM Reasoning

arXiv: https://arxiv.org/abs/2504.18587 · HTML: https://mlsanity.com/paper/2504.18587/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [When More is Less: Understanding Chain-of-Thought Length in LLMs](https://mlsanity.com/paper/2502.07266/) (2025)
- [Qwen2.5-1M Technical Report](https://mlsanity.com/paper/2501.15383/) (2025)
- [T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling](https://mlsanity.com/paper/2501.11651/) (2025)

## Built on by (most cited first, 3 of 3)

- [Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2505.17826/) (2025)
- [Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning](https://mlsanity.com/paper/2508.04848/) (2025)
- [Learning to Reason in LLMs by Expectation Maximization](https://mlsanity.com/paper/2512.20169/) (2025)

## Abstract (arXiv)

Reinforcement learning has emerged as a powerful approach for improving the reasoning capabilities of large language models, as demonstrated by systems such as OpenAI's O1~citeo1 and DeepSeek-R1~citer1. However, widely used algorithms such as PPO~citeppo and GRPO~citegrpo rely on likelihood-ratio correction and heuristic clipping, which increase algorithmic complexity and may introduce additional variance and optimization bias. We present EM Policy Gradient (EMPG), a simple and principled framework that formulates reinforcement learning over reasoning trajectories as an Expectation-Maximization (EM) problem~citeem, dempster1977em. The E-step constructs a reward-weighted distribution over rationale trajectories, while the M-step improves the language model by maximizing their weighted log-likelihood. This formulation leads to a practical off-policy policy-gradient algorithm that directly optimizes the EM surrogate objective without trajectory-level importance weighting or likelihood-ratio clipping. Moreover, under exact E- and improving M-steps, EMPG guarantees monotonic improvement of the expected-reward objective. Experiments on GSM8K and MATH Hard with multiple Qwen2.5 base models show that EMPG achieves performance comparable to or better than GRPO while using a substantially simpler optimization procedure. EMPG also produces more concise reasoning trajectories and exhibits structured reasoning behaviors, including subproblem decomposition, self-verification, reflection, and backtracking. These results suggest that the EM perspective provides an effective theoretical and algorithmic foundation for reinforcement learning of reasoning language models.