# Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning

arXiv: https://arxiv.org/abs/2604.27998 · HTML: https://mlsanity.com/paper/2604.27998/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step](https://mlsanity.com/paper/2405.14838/) (2024)
- [Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space](https://mlsanity.com/paper/2505.15778/) (2025)
- [Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains](https://mlsanity.com/paper/2505.16552/) (2025)
- [LLM Latent Reasoning as Chain of Superposition](https://mlsanity.com/paper/2510.15522/) (2025)
- [LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking](https://mlsanity.com/paper/2508.03440/) (2025)
- [Soft Tokens, Hard Truths](https://mlsanity.com/paper/2509.19170/) (2025)
- [Text Generation Beyond Discrete Token Sampling](https://mlsanity.com/paper/2505.14827/) (2025)
- [SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization](https://mlsanity.com/paper/2511.06411/) (2025)
- [Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning](https://mlsanity.com/paper/2601.14750/) (2026)
- [The Evolution of Thought: Tracking LLM Overthinking via Reasoning Dynamics Analysis](https://mlsanity.com/paper/2508.17627/) (2025)
- [Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge](https://mlsanity.com/paper/2601.08808/) (2026)
- [Parallel Test-Time Scaling for Latent Reasoning Models](https://mlsanity.com/paper/2510.07745/) (2025)
- [CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions](https://mlsanity.com/paper/2602.20213/) (2026)
- [Circular Reasoning: Understanding Self-Reinforcing Loops in Large Reasoning Models](https://mlsanity.com/paper/2601.05693/) (2026)
- [Projecting Out the Malice: A Global Subspace Approach to LLM Detoxification](https://mlsanity.com/paper/2601.06226/) (2026)
- [See or Say Graphs: Agent-Driven Scalable Graph Structure Understanding with Vision-Language Models](https://mlsanity.com/paper/2510.16769/) (2025)

## Built on by (most cited first, 10 of 10)

- [Solve the Loop: Attractor Models for Language and Reasoning](https://mlsanity.com/paper/2605.12466/) (2026)
- [Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning](https://mlsanity.com/paper/2606.13106/) (2026)
- [ESPO: Early-Stopping Proximal Policy Optimization](https://mlsanity.com/paper/2605.29860/) (2026)
- [RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection](https://mlsanity.com/paper/2608.01630/) (2026)
- [Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery](https://mlsanity.com/paper/2609.19164/) (2026)
- [AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models](https://mlsanity.com/paper/2610.01560/) (2026)
- [TARPO: Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization](https://mlsanity.com/paper/2606.05859/) (2026)
- [SLPO: Scaling Latent Reasoning via a Surrogate Policy](https://mlsanity.com/paper/2607.19691/) (2026)
- [SkillFM: Generating Skills for LLM Agents via Latent Flow Matching](https://mlsanity.com/paper/2609.39382/) (2026)
- [Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression](https://mlsanity.com/paper/2610.09788/) (2026)

## Abstract (arXiv)

Latent reasoning offers a more efficient alternative to explicit reasoning by compressing intermediate reasoning into continuous representations and substantially shortening reasoning chains. However, existing latent reasoning methods mainly focus on supervised learning, and reinforcement learning in latent space remains highly unstable. We study this problem through the lens of Group Relative Policy Optimization (GRPO), and show that directly adapting GRPO to latent reasoning is fundamentally non-trivial: latent reasoning changes both the probability density and the sampling mechanism, causing three coupled bottlenecks: absence of intrinsic latent manifolds, where unconstrained exploration pushes rollouts off the valid latent manifold; exploration-optimization misalignment, where trajectory-level rewards can induce incorrect token-level updates; and latent mixture non-closure, where jointly reinforcing multiple correct latent paths can produce an invalid averaged state. To address them, we propose Latent-GRPO, which combines invalid-sample advantage masking, one-sided noise sampling, and optimal correct-path first-token selection. Across four low-difficulty benchmarks (e.g., GSM8K-Aug) and four high-difficulty benchmarks (e.g., AIME), Latent-GRPO improves over its latent initialization by 7.86 Pass@1 points on low-difficulty tasks and surpasses explicit GRPO by 4.27 points on high-difficulty tasks while using 3--4× shorter reasoning chains. It also achieves stronger pass@k performance under Gumbel sampling. These results establish Latent-GRPO as an effective approach for stable and efficient latent reasoning.