# Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding

arXiv: https://arxiv.org/abs/2411.04282 · HTML: https://mlsanity.com/paper/2411.04282/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 28 of its 56 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 28 of 28)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [Chain-of-Thought Reasoning Without Prompting](https://mlsanity.com/paper/2402.10200/) (2024)
- [Teaching Large Language Models to Reason with Reinforcement Learning](https://mlsanity.com/paper/2403.04642/) (2024)
- [Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization](https://mlsanity.com/paper/2308.02151/) (2023)
- [Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs](https://mlsanity.com/paper/2402.14740/) (2024)
- [Amortizing intractable inference in large language models](https://mlsanity.com/paper/2310.04363/) (2023)
- [Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents](https://mlsanity.com/paper/2408.07060/) (2024)

## Built on by (most cited first, 30 of 55)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [Token-Budget-Aware LLM Reasoning](https://mlsanity.com/paper/2412.18547/) (2024)
- [STAIR: Improving Safety Alignment with Introspective Reasoning](https://mlsanity.com/paper/2502.02384/) (2025)
- [Reinforcing General Reasoning without Verifiers](https://mlsanity.com/paper/2505.21493/) (2025)
- [Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens](https://mlsanity.com/paper/2511.19418/) (2025)
- [Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendation](https://mlsanity.com/paper/2503.22675/) (2025)
- [Reasoning to Learn from Latent Thoughts](https://mlsanity.com/paper/2503.18866/) (2025)
- [Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space](https://mlsanity.com/paper/2505.13308/) (2025)
- [Implicit Reasoning in Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.02350/) (2025)
- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops](https://mlsanity.com/paper/2607.07663/) (2026)
- [Learning to Reason for Long-Form Story Generation](https://mlsanity.com/paper/2503.22828/) (2025)
- [MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning](https://mlsanity.com/paper/2505.24871/) (2025)
- [Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics](https://mlsanity.com/paper/2504.01995/) (2025)
- [Activation Steering for Chain-of-Thought Compression](https://mlsanity.com/paper/2507.04742/) (2025)
- [Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks](https://mlsanity.com/paper/2506.13351/) (2025)
- [Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty](https://mlsanity.com/paper/2506.10446/) (2025)
- [Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers](https://mlsanity.com/paper/2505.20128/) (2025)
- [BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning](https://mlsanity.com/paper/2501.18858/) (2025)
- [VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reduction](https://mlsanity.com/paper/2602.12579/) (2026)
- [Behavior Injection: Preparing Language Models for Reinforcement Learning](https://mlsanity.com/paper/2505.18917/) (2025)
- [Variational Reasoning for Language Models](https://mlsanity.com/paper/2509.22637/) (2025)
- [Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels](https://mlsanity.com/paper/2510.06499/) (2025)
- [LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoning](https://mlsanity.com/paper/2602.07075/) (2026)
- [Efficient Post-Training Refinement of Latent Reasoning in Large Language Models](https://mlsanity.com/paper/2506.08552/) (2025)
- [SSR: Socratic Self-Refine for Large Language Model Reasoning](https://mlsanity.com/paper/2511.10621/) (2025)
- [Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering](https://mlsanity.com/paper/2601.08427/) (2026)
- [From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignment](https://mlsanity.com/paper/2503.20472/) (2025)
- [CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards](https://mlsanity.com/paper/2507.17147/) (2025)
- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown impressive capabilities, but still struggle with complex reasoning tasks requiring multiple steps. While prompt-based methods like Chain-of-Thought (CoT) can improve LLM reasoning at inference time, optimizing reasoning capabilities during training remains challenging. We introduce LaTent Reasoning Optimization (LaTRO), a principled framework that formulates reasoning as sampling from a latent distribution and optimizes it via variational approaches. LaTRO enables LLMs to concurrently improve both their reasoning process and ability to evaluate reasoning quality, without requiring external feedback or reward models. We validate LaTRO through experiments on GSM8K and ARC-Challenge datasets using multiple model architectures. On GSM8K, LaTRO improves zero-shot accuracy by an average of 12.5% over base models and 9.6% over supervised fine-tuning across Phi-3.5-mini, Mistral-7B, and Llama-3.1-8B. Our findings suggest that pre-trained LLMs possess latent reasoning capabilities that can be unlocked and enhanced through our proposed optimization approach in a self-improvement manner. The code of LaTRO is available at urlhttps://github.com/SalesforceAIResearch/LaTRO.