An Expectation-Maximization Perspective on Reinforcement Learning for LLM Reasoning

Reinforcement learning has emerged as a powerful approach for improving the reasoning capabilities of large language models, as demonstrated by systems such as OpenAI's O1~citeo1 and DeepSeek-R1~citer1. However, widely used algorithms such as PPO~citeppo and GRPO~citegrpo rely on likelihood-ratio correction and heuristic clipping, which increase algorithmic complexity and may introduce additional variance and optimization bias. We present EM Policy Gradient (EMPG), a simple and principled framework that formulates reinforcement learning over reasoning trajectories as an Expectation-Maximization (EM) problem~citeem, dempster1977em. The E-step constructs a reward-weighted distribution over rationale trajectories, while the M-step improves the language model by maximizing their weighted log-likelihood. This formulation leads to a practical off-policy policy-gradient algorithm that directly optimizes the EM surrogate objective without trajectory-level importance weighting or likelihood-ratio clipping. Moreover, under exact E- and improving M-steps, EMPG guarantees monotonic improvement of the expected-reward objective. Experiments on GSM8K and MATH Hard with multiple Qwen2.5 base models show that EMPG achieves performance comparable to or better than GRPO while using a substantially simpler optimization procedure. EMPG also produces more concise reasoning trajectories and exhibits structured reasoning behaviors, including subproblem decomposition, self-verification, reflection, and backtracking. These results suggest that the EM perspective provides an effective theoretical and algorithmic foundation for reinforcement learning of reasoning language models.