# Parallel Continuous Chain-of-Thought with Jacobi Iteration

arXiv: https://arxiv.org/abs/2506.18582 · HTML: https://mlsanity.com/paper/2506.18582/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 18 of 18)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [Think before you speak: Training Language Models With Pause Tokens](https://mlsanity.com/paper/2310.02226/) (2023)
- [Reasoning with Latent Thoughts: On the Power of Looped Transformers](https://mlsanity.com/paper/2502.17416/) (2025)
- [From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step](https://mlsanity.com/paper/2405.14838/) (2024)
- [CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation](https://mlsanity.com/paper/2502.21074/) (2025)
- [Compressed Chain of Thought: Efficient Reasoning Through Dense Representations](https://mlsanity.com/paper/2412.13171/) (2024)
- [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://mlsanity.com/paper/2309.15402/) (2023)
- [CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text](https://mlsanity.com/paper/1908.06177/) (2019)
- [Accelerating Transformer Inference for Translation via Parallel Decoding](https://mlsanity.com/paper/2305.10427/) (2023)
- [SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs](https://mlsanity.com/paper/2502.12134/) (2025)
- [Efficient Inference for Large Reasoning Models: A Survey](https://mlsanity.com/paper/2503.23077/) (2025)
- [Layer-Condensed KV Cache for Efficient Inference of Large Language Models](https://mlsanity.com/paper/2405.10637/) (2024)

## Built on by (most cited first, 25 of 25)

- [A Survey on Latent Reasoning](https://mlsanity.com/paper/2507.06203/) (2025)
- [LLM Latent Reasoning as Chain of Superposition](https://mlsanity.com/paper/2510.15522/) (2025)
- [SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs](https://mlsanity.com/paper/2510.05069/) (2025)
- [Soft-Masked Diffusion Language Models](https://mlsanity.com/paper/2510.17206/) (2025)
- [Parallel Test-Time Scaling for Latent Reasoning Models](https://mlsanity.com/paper/2510.07745/) (2025)
- [Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure](https://mlsanity.com/paper/2602.08783/) (2026)
- [KaVa: Latent Reasoning via Compressed KV-Cache Distillation](https://mlsanity.com/paper/2510.02312/) (2025)
- [LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoning](https://mlsanity.com/paper/2602.07075/) (2026)
- [SeLaR: Selective Latent Reasoning in Large Language Models](https://mlsanity.com/paper/2604.08299/) (2026)
- [T^2MLR: Transformer with Temporal Middle-Layer Recurrence](https://mlsanity.com/paper/2607.15178/) (2026)
- [Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior](https://mlsanity.com/paper/2605.26797/) (2026)
- [nabla-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space](https://mlsanity.com/paper/2603.04948/) (2026)
- [What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis](https://mlsanity.com/paper/2606.20075/) (2026)
- [Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression](https://mlsanity.com/paper/2606.29712/) (2026)
- [Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoning](https://mlsanity.com/paper/2606.02842/) (2026)
- [A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM](https://mlsanity.com/paper/2609.07821/) (2026)
- [Assessing Adversarial Robustness of Latent Reasoning Models](https://mlsanity.com/paper/2609.22228/) (2026)
- [AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models](https://mlsanity.com/paper/2610.01560/) (2026)
- [LoRi: Low-Rank Distillation for Implicit Reasoning](https://mlsanity.com/paper/2606.05315/) (2026)
- [Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Models](https://mlsanity.com/paper/2609.15338/) (2026)
- [Pretraining Latent Information Feedback Transformers with Teacher Supervision](https://mlsanity.com/paper/2609.38149/) (2026)
- [Plan Canvas: Fixed Reasoning Regions for Continuous Language Flows](https://mlsanity.com/paper/2610.05815/) (2026)
- [RUBATO: Decoupled Clocks Allow Full-Duplex Models to Think as Fast as They Can](https://mlsanity.com/paper/2610.14299/) (2026)
- [Dynamic K-Head: Difficulty-Supervised Budget Selection for Continuous Latent Reasoning](https://mlsanity.com/paper/2610.17679/) (2026)
- [Rationale Distribution Distillation via Flow Matching for Distributional Continuous Latent Reasoning](https://mlsanity.com/paper/2610.19871/) (2026)

## Abstract (arXiv)

Continuous chain-of-thought has been shown to be effective in saving reasoning tokens for large language models. By reasoning with continuous latent thought tokens, continuous CoT is able to perform implicit reasoning in a compact manner. However, the sequential dependencies between latent thought tokens spoil parallel training, leading to long training time. In this paper, we propose Parallel Continuous Chain-of-Thought (PCCoT), which performs Jacobi iteration on the latent thought tokens, updating them iteratively in parallel instead of sequentially and thus improving both training and inference efficiency of continuous CoT. Experiments demonstrate that by choosing the proper number of iterations, we are able to achieve comparable or even better performance while saving nearly 50% of the training and inference time. Moreover, PCCoT shows better stability and robustness in the training process. Our code is available at https://github.com/whyNLP/PCCoT.