# A Survey on Parallel Reasoning

arXiv: https://arxiv.org/abs/2510.12164 · HTML: https://mlsanity.com/paper/2510.12164/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 61 of its 118 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 61)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [Deep Think with Confidence](https://mlsanity.com/paper/2508.15260/) (2025)
- [An LLM Compiler for Parallel Function Calling](https://mlsanity.com/paper/2312.04511/) (2023)
- [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022)
- [Accelerating LLM Inference with Staged Speculative Decoding](https://mlsanity.com/paper/2308.04623/) (2023)
- [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023)
- [Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding](https://mlsanity.com/paper/2402.05109/) (2024)
- [Cascade Speculative Drafting for Even Faster LLM Inference](https://mlsanity.com/paper/2312.11462/) (2023)
- [MALT: Improving Reasoning with Multi-Agent LLM Training](https://mlsanity.com/paper/2412.01928/) (2024)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration](https://mlsanity.com/paper/2410.06916/) (2024)
- [Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration](https://mlsanity.com/paper/2310.00280/) (2023)
- [Parallel-R1: Towards Parallel Thinking via Reinforcement Learning](https://mlsanity.com/paper/2509.07980/) (2025)
- [Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting](https://mlsanity.com/paper/2404.18911/) (2024)

## Built on by (most cited first, 11 of 11)

- [ParallelMuse: Agentic Parallel Thinking for Deep Information Seeking](https://mlsanity.com/paper/2510.24698/) (2025)
- [Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning](https://mlsanity.com/paper/2512.07461/) (2025)
- [Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution](https://mlsanity.com/paper/2601.20379/) (2026)
- [CAPS: Cascaded Adaptive Pairwise Selection for Efficient Parallel Reasoning](https://mlsanity.com/paper/2605.15513/) (2026)
- [Share More, Search Less: Collaborative Parallel Thinking for Efficient Test-Time Scaling](https://mlsanity.com/paper/2605.27030/) (2026)
- [MedVerse: Efficient and Reliable Medical Reasoning via DAG-Structured Parallel Execution](https://mlsanity.com/paper/2602.07529/) (2026)
- [Asking the Right Questions: Improving Reasoning with Generated Stepping Stones](https://mlsanity.com/paper/2602.19069/) (2026)
- [OPE: Overcoming Information Saturation in Parallel Thinking via Outline-Guided Path Exploration](https://mlsanity.com/paper/2602.08344/) (2026)
- [Cut Your Losses! Learning to Prune Paths Early for Efficient Parallel Reasoning](https://mlsanity.com/paper/2604.16029/) (2026)
- [Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning](https://mlsanity.com/paper/2607.12395/) (2026)
- [Parason: Revealing Subtask and Trial Parallelism in LLM Reasoning](https://mlsanity.com/paper/2608.24658/) (2026)

## Abstract (arXiv)

With the increasing capabilities of Large Language Models (LLMs), parallel reasoning has emerged as a new inference paradigm that enhances reasoning robustness by concurrently exploring multiple lines of thought before converging on a final answer. It has become a significant trend to explore parallel reasoning to overcome the fragility of standard sequential methods and improve practical performance. In this paper, we aim to survey and summarize the progress and challenges of parallel reasoning. We first present a formal definition of parallel reasoning and clarify its distinction from related concepts like Chain-of-Thought. Then, we organize and discuss advanced techniques based on a novel taxonomy, including non-interactive reasoning, interactive reasoning, and efficiency-focused decoding strategies. Additionally, we explore various application scenarios, such as solving complex problems and enhancing the reliability of LLM outputs.Finally, we highlight the core challenges of parallel reasoning and suggest potential directions for future research. We hope that our work can provide a useful roadmap for beginners and encourage more research on improving parallel reasoning methods. Related source can be avaliable in https://github.com/PPPP-kaqiu/Awesome-Parallel-Reasoning.