mlsanityData to 7 Oct 2026

Learning to Reason in LLMs by Expectation Maximization

arXiv 2512.20169 ↗First posted 23 Dec 2025cs.LG cs.CL stat.MLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) solve reasoning problems by first generating a rationale and then answering. We formalize reasoning as a latent variable model and derive a reward-based filtered expectation-maximization (FEM) objective for learning to reason. This view connects EM and modern reward-based optimization, and shows that the main challenge lies in designing a sampling distribution of rationales that justify correct answers. We instantiate and compare three sampling schemes: rejection sampling with a budget, self-taught reasoner (STaR), and prompt posterior sampling (PPS), which only keeps the rationalization stage of STaR that conditions on the correct answer in the prompt. We experiment with LLM-as-a-judge calibration and summarization from feedback tasks, where conditioning on the correct answer provides a strong guidance for generating rationales. Our experiments show the efficacy of PPS over other sampling schemes, and that the sampling scheme can have a significant impact on performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 53 of its 76 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

53 of 76 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners22 May
  2. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning31 Jan
  3. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding6 Nov
  4. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2024Qwen2.5 Technical Reportcited by 4,713
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
37 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  9. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  11. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  13. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  14. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  15. 2021GFlowNet Foundationscited by 286
  16. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  17. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  18. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  19. 2014Statistical guarantees for the EM algorithm: From population to sample-based analysiscited by 131
  20. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  21. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  22. 2020A Distributional Approach to Controlled Text Generationcited by 117
  23. 2024Critique-out-Loud Reward Modelscited by 105
  24. 2023Amortizing intractable inference in large language modelscited by 94
  25. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  26. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 84
  27. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  28. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 46
  29. 2025Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Datacited by 29
  30. 2019Imitation-Regularized Offline Learningcited by 17
  31. 2025TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judgecited by 16
  32. 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10
  33. 2022Local Policy Improvement for Recommender Systemscited by 8
  34. 2024Guaranteed Generation from Large Language Modelscited by 7
  35. 2025Quantitative LLM Judgescited by 6
  36. 2025An Expectation-Maximization Perspective on Reinforcement Learning for LLM Reasoningcited by 3
  37. 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversitycited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocation
  2. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  3. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
  4. 2026Adaptive Power Sampling for LLM Reasoning
  5. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
  6. 2025The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
  7. 2025TextBandit: Evaluating Probabilistic Reasoning in LLMs Through Language-Only Decision Tasks
  8. 2026Learning to Reason Efficiently with A* Post-Training