# MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelining Framework for Prefill-Only LLM Inference

arXiv: https://arxiv.org/abs/2606.22968 · HTML: https://mlsanity.com/paper/2606.22968/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 18 of 18)

- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models](https://mlsanity.com/paper/2102.07988/) (2021)
- [ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale](https://mlsanity.com/paper/2303.14006/) (2023)
- [PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications](https://mlsanity.com/paper/2505.07203/) (2025)
- [WaferLLM: Large Language Model Inference at Wafer Scale](https://mlsanity.com/paper/2502.04563/) (2025)
- [SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling](https://mlsanity.com/paper/2407.10416/) (2024)
- [Wafer-scale Computing: Advancements, Challenges, and Future Perspectives](https://mlsanity.com/paper/2310.09568/) (2023)
- [MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness](https://mlsanity.com/paper/2509.10372/) (2025)
- [MoEntwine: Unleashing the Potential of Wafer-scale Chips for Large-scale Expert Parallel Inference](https://mlsanity.com/paper/2510.25258/) (2025)
- [Ouroboros: Wafer-Scale SRAM CIM with Token-Grained Pipelining for Large Language Model Inference](https://mlsanity.com/paper/2603.02737/) (2026)
- [WATOS: Efficient LLM Training Strategies and Architecture Co-exploration for Wafer-scale Chip](https://mlsanity.com/paper/2512.12279/) (2025)
- [PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion](https://mlsanity.com/paper/2512.14322/) (2025)
- [PALM: A Efficient Performance Simulator for Tiled Accelerators with Large-scale Model Training](https://mlsanity.com/paper/2406.03868/) (2024)
- [FRED: Flexible REduction-Distribution Interconnect and Communication Implementation for Wafer-Scale Distributed Training of DNN Models](https://mlsanity.com/paper/2406.19580/) (2024)
- [BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Termination](https://mlsanity.com/paper/2512.06457/) (2025)
- [LAPA: Log-Domain Prediction-Driven Dynamic Sparsity Accelerator for Transformer Model](https://mlsanity.com/paper/2512.07855/) (2025)
- [TEMP: A Memory Efficient Physical-aware Tensor Partition-Mapping Framework on Wafer-scale Chips](https://mlsanity.com/paper/2512.14256/) (2025)
- [Designing Spatial Architectures for Sparse Attention: STAR Accelerator via Cross-Stage Tiling](https://mlsanity.com/paper/2512.20198/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used in prefill-only workloads, where end-to-end latency is dominated by the prefill phase. For long-context prefill, communication overhead grows with sequence length and quickly becomes a bottleneck on conventional GPU systems, making wafer-scale chips (WSCs) a promising substrate due to their high communication bandwidth and large aggregate compute and memory capacity. A natural way to accelerate prefill is to partition a long input sequence into multiple chunks and execute them in a finer-grained pipeline across devices. However, directly applying this idea to long-context prefill on WSCs remains challenging. First, causal dependency across chunks causes KV cache to accumulate unevenly across pipeline stages, creating severe memory imbalance and limiting the feasible sequence length. Second, later chunks require more attention computation because each chunk depends on preceding chunks, leading to chunk-level latency imbalance. To address these challenges, we present MOCAP, a memory-orchestrated chunked pipelining framework for prefill-only LLM inference on WSCs. MOCAP introduces Memory-Balanced KV Reallocation (MBKR) to alleviate memory imbalance by redistributing KV cache across pipeline stages, thereby extending the feasible sequence length. It further incorporates Latency-Balanced Chunk Partitioning (LBCP) to balance chunk execution cost under both attention-cost growth and KV reallocation overhead, improving pipeline efficiency. Experimental results show that, compared with GPipe, MOCAP achieves 76.4% lower end-to-end latency and 3.24× higher throughput on average. MOCAP also extends the maximum supported sequence length by up to 1.31× compared with Terapipe.