# TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference

arXiv: https://arxiv.org/abs/2506.10470 · HTML: https://mlsanity.com/paper/2506.10470/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2025.

## Built on by (most cited first, 8 of 8)

- [Towards Resiliency in Large Language Model Serving with KevlarFlow](https://mlsanity.com/paper/2601.22438/) (2026)
- [Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets](https://mlsanity.com/paper/2608.19147/) (2026)
- [Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications](https://mlsanity.com/paper/2601.08833/) (2025)
- [NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining](https://mlsanity.com/paper/2604.06956/) (2026)
- [PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers](https://mlsanity.com/paper/2605.02189/) (2026)
- [KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMs](https://mlsanity.com/paper/2608.15943/) (2026)
- [WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse](https://mlsanity.com/paper/2609.35263/) (2026)
- [Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure](https://mlsanity.com/paper/2609.38697/) (2026)

## Abstract (arXiv)

As the model size continuously increases, pipeline parallelism shows great promise in throughput-oriented LLM inference due to its low demand on communications. However, imbalanced pipeline workloads and complex data dependencies in the prefill and decode phases result in massive pipeline bubbles and further severe performance reduction. To better exploit the pipeline parallelism for high-throughput LLM inference, we propose TD-Pipe, with the key idea lies in the temporally-disaggregated pipeline parallelism architecture. Specifically, this architecture disaggregates the prefill and decode phases in the temporal dimension, so as to eliminate pipeline bubbles caused by the phase switching. TD-Pipe identifies potential issues of exploiting the novel architecture and provides solutions. First, a hierarchy-controller structure is used to better coordinate devices in pipeline parallelism by decoupling the scheduling from execution. Second, the AI-based greedy prefill approach aggressively performs more prefills by predicting the output length and simulating the memory usage. Third, the inter-batch work stealing approach dynamically balances decode phase workloads between different batches to reduce bubbles. Forth, the spatial-temporal intensity comparison approach determines the optimal switch from decode to prefill by comparing the performance drop from reduced computational intensity with that from phase switching bubbles. Extensive experiments show that TD-Pipe effectively increases the throughput of LLM inference by up to 1.91x over the existing tensor parallel approach and 2.73x over the existing pipeline parallel approach on GPU nodes with only PCIe interconnection.