# ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute

arXiv: https://arxiv.org/abs/2509.04475 · HTML: https://mlsanity.com/paper/2509.04475/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 37 of its 54 references that are in the map.
- First posted on arXiv 30 Aug 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 37)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://mlsanity.com/paper/2501.04519/) (2025)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [Remasking Discrete Diffusion Models with Inference-Time Scaling](https://mlsanity.com/paper/2503.00307/) (2025)
- [d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning](https://mlsanity.com/paper/2504.12216/) (2025)
- [The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks](https://mlsanity.com/paper/2502.08235/) (2025)
- [Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.07572/) (2025)
- [RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?](https://mlsanity.com/paper/2501.11284/) (2025)
- [Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?](https://mlsanity.com/paper/2504.06514/) (2025)
- [Learning Adaptive Parallel Reasoning with Language Models](https://mlsanity.com/paper/2504.15466/) (2025)
- [Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models](https://mlsanity.com/paper/2506.04210/) (2025)
- [SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoning](https://mlsanity.com/paper/2505.11274/) (2025)
- [Theoretical Benefit and Limitation of Diffusion Language Model](https://mlsanity.com/paper/2502.09622/) (2025)
- [Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning](https://mlsanity.com/paper/2308.12219/) (2023)

## Built on by (most cited first, 30 of 38)

- [Parallel-Probe: Towards Efficient Parallel Thinking via 2D Probing](https://mlsanity.com/paper/2602.03845/) (2026)
- [Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models](https://mlsanity.com/paper/2509.26628/) (2025)
- [Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning](https://mlsanity.com/paper/2512.07461/) (2025)
- [A Survey on Parallel Reasoning](https://mlsanity.com/paper/2510.12164/) (2025)
- [Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?](https://mlsanity.com/paper/2602.23225/) (2026)
- [LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling](https://mlsanity.com/paper/2605.08083/) (2026)
- [Planned Diffusion](https://mlsanity.com/paper/2510.18087/) (2025)
- [Thinking with Map: Reinforced Parallel Map-Augmented Agent for Geolocalization](https://mlsanity.com/paper/2601.05432/) (2026)
- [ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models](https://mlsanity.com/paper/2512.07843/) (2025)
- [LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoning](https://mlsanity.com/paper/2602.07075/) (2026)
- [The Era of Agentic Organization: Learning to Organize with Language Models](https://mlsanity.com/paper/2510.26658/) (2025)
- [Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization](https://mlsanity.com/paper/2601.01483/) (2026)
- [Training Large Language Models To Reason In Parallel With Global Forking Tokens](https://mlsanity.com/paper/2510.05132/) (2025)
- [DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing](https://mlsanity.com/paper/2601.09609/) (2026)
- [Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving](https://mlsanity.com/paper/2603.07197/) (2026)
- [InfoSeeker: A Scalable Hierarchical Parallel Agent Framework for Web Information Seeking](https://mlsanity.com/paper/2604.02971/) (2026)
- [Efficient Tree-Structured Deep Research with Adaptive Resource Allocation](https://mlsanity.com/paper/2510.05145/) (2025)
- [Test-Time Compute Games](https://mlsanity.com/paper/2601.21839/) (2026)
- [Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Models](https://mlsanity.com/paper/2604.05868/) (2026)
- [Compute Allocation for Self-Evolving LLMs: From Depth-Breadth to Multi-Armed Bandits](https://mlsanity.com/paper/2605.29268/) (2026)
- [Limited Reasoning Space: The cage of long-horizon reasoning in LLMs](https://mlsanity.com/paper/2602.19281/) (2026)
- [Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs](https://mlsanity.com/paper/2605.12460/) (2026)
- [OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation](https://mlsanity.com/paper/2605.15177/) (2026)
- [SPIRAL: Learning to Search and Aggregate](https://mlsanity.com/paper/2606.23595/) (2026)
- [On the Overscaling Curse of Parallel Thinking: System Efficacy Contradicts Sample Efficiency](https://mlsanity.com/paper/2601.21619/) (2026)
- [Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning](https://mlsanity.com/paper/2601.23027/) (2026)
- [One-Token Verification for Reasoning Correctness Estimation](https://mlsanity.com/paper/2603.01025/) (2026)
- [LaneRoPE: Positional Encoding for Collaborative Parallel Reasoning and Generation](https://mlsanity.com/paper/2605.27570/) (2026)
- [Visual Para-Thinker++: A Single-Policy Multi-Agent Framework for Visual Reasoning](https://mlsanity.com/paper/2606.09290/) (2026)
- [Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents](https://mlsanity.com/paper/2610.07447/) (2026)

## Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have been driven by test-time compute scaling - a strategy that improves reasoning by generating longer, sequential thought processes. While effective, this approach encounters a significant bottleneck as computation increases, where further computation offers only marginal performance gains. We argue this ceiling is not an inherent limit of the model's capability but a flaw in the scaling strategy itself, a phenomenon we term "Tunnel Vision", where a model's imperfect initial steps lock it into a suboptimal reasoning path. To overcome this, we introduce a new scaling paradigm: native thought parallelism. We present ParaThinker, an end-to-end framework that trains an LLM to generate multiple, diverse reasoning paths in parallel and synthesize them into a superior final answer. By exploring different lines of thoughts simultaneously, ParaThinker effectively sidesteps the Tunnel Vision issue and unlocks the model's latent reasoning potential. Our approach demonstrates that scaling compute in parallel (width) is a more effective and efficient way to superior reasoning than simply scaling sequentially (depth). On challenging reasoning benchmarks, ParaThinker achieves substantial accuracy improvements over sequential LLMs (12.3% for 1.5B and 7.5% for 7B models on average with 8 parallel paths), while adding only negligible latency overhead (7.1%). This enables smaller models to surpass much larger counterparts and establishes parallel thinking as a critical, efficient dimension for scaling future LLMs.