# Hybrid Latent Reasoning via Reinforcement Learning

arXiv: https://arxiv.org/abs/2505.18454 · HTML: https://mlsanity.com/paper/2505.18454/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 44 of its 53 references that are in the map.
- First posted on arXiv 24 May 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 44)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Think before you speak: Training Language Models With Pause Tokens](https://mlsanity.com/paper/2310.02226/) (2023)
- [Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study](https://mlsanity.com/paper/2404.10719/) (2024)
- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)

## Built on by (most cited first, 30 of 32)

- [Dr. Zero: Self-Evolving Search Agents without Training Data](https://mlsanity.com/paper/2601.07055/) (2026)
- [Implicit Reasoning in Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.02350/) (2025)
- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [Continuous Chain of Thought Enables Parallel Exploration and Reasoning](https://mlsanity.com/paper/2505.23648/) (2025)
- [SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs](https://mlsanity.com/paper/2510.05069/) (2025)
- [Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning](https://mlsanity.com/paper/2601.14750/) (2026)
- [Generative Recursive Reasoning](https://mlsanity.com/paper/2605.19376/) (2026)
- [Mull-Tokens: Modality-Agnostic Latent Thinking](https://mlsanity.com/paper/2512.10941/) (2025)
- [Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models](https://mlsanity.com/paper/2602.10520/) (2026)
- [ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces](https://mlsanity.com/paper/2602.11683/) (2026)
- [T^2MLR: Transformer with Temporal Middle-Layer Recurrence](https://mlsanity.com/paper/2607.15178/) (2026)
- [Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout](https://mlsanity.com/paper/2605.05092/) (2026)
- [Full-bandwidth transformer](https://mlsanity.com/paper/2608.08888/) (2026)
- [LEPO: Latent Reasoning Policy Optimization for Large Language Models](https://mlsanity.com/paper/2604.17892/) (2026)
- [Retrieval Augmented Conversational Recommendation with Reinforcement Learning](https://mlsanity.com/paper/2604.04457/) (2026)
- [TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens](https://mlsanity.com/paper/2605.16638/) (2026)
- [Latent Thought Flow: Efficient Latent Reasoning in Large Language Models](https://mlsanity.com/paper/2606.16222/) (2026)
- [Latent Visual Cache for Video Reasoning](https://mlsanity.com/paper/2607.02607/) (2026)
- [Improving Latent Reasoning in LLMs via Soft Concept Mixing](https://mlsanity.com/paper/2511.16885/) (2025)
- [Training Continuous Chain of Thought Models: A Tale of Two Regimes](https://mlsanity.com/paper/2607.16972/) (2026)
- [Canvas-of-Thought: Grounding Reasoning via Mutable Structured States](https://mlsanity.com/paper/2602.10494/) (2026)
- [Latent Abstraction for Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.17866/) (2026)
- [Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance](https://mlsanity.com/paper/2604.23318/) (2026)
- [GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understanding](https://mlsanity.com/paper/2607.13454/) (2026)
- [Multi-Path Collaborative Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2512.01485/) (2025)
- [Geometric Latent Reasoning Induces Shorter Generations in LLMs](https://mlsanity.com/paper/2606.02248/) (2026)
- [Dropout-GRPO: Variational Stochasticity for Continuous Latent Reasoning](https://mlsanity.com/paper/2606.10184/) (2026)
- [N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization](https://mlsanity.com/paper/2606.10768/) (2026)
- [Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering](https://mlsanity.com/paper/2608.04124/) (2026)
- [GenoMorph: Pathway-Grounded Genomic Disease Reasoning via Adaptive Latent Computation](https://mlsanity.com/paper/2609.34079/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have introduced latent reasoning as a promising alternative to autoregressive reasoning. By performing internal computation with hidden states from previous steps, latent reasoning benefit from more informative features rather than sampling a discrete chain-of-thought (CoT) path. Yet latent reasoning approaches are often incompatible with LLMs, as their continuous paradigm conflicts with the discrete nature of autoregressive generation. Moreover, these methods rely on CoT traces for training and thus fail to exploit the inherent reasoning patterns of LLMs. In this work, we explore latent reasoning by leveraging the intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid latent reasoning approach that (1) integrates prior hidden states into sampled tokens with a learnable gating mechanism, and (2) initializes training with predominantly token embeddings while progressively incorporating more hidden features. This design maintains LLMs' generative capabilities and incentivizes hybrid reasoning using both discrete and continuous representations. In addition, the hybrid HRPO introduces stochasticity into latent reasoning via token sampling, thereby enabling RL-based optimization without requiring CoT trajectories. Extensive evaluations across diverse benchmarks show that HRPO outperforms prior methods in both knowledge- and reasoning-intensive tasks. Furthermore, HRPO-trained LLMs remain interpretable and exhibit intriguing behaviors like cross-lingual patterns and shorter completion lengths, highlighting the potential of our RL-based approach and offer insights for future work in latent reasoning.