mlsanityData to 14 Oct 2026

SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference

arXiv 2510.08544 ↗First posted 9 Oct 2025cs.AR cs.DC cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have gained popularity in recent years, driving up the demand for inference. LLM inference is composed of two phases with distinct characteristics: a compute-bound prefill phase followed by a memory-bound decode phase. To efficiently serve LLMs, prior work proposes prefill-decode disaggregation to run each phase on separate hardware. However, existing hardware poorly matches the different requirements of each phase. Current datacenter GPUs and TPUs follow a more-is-better design philosophy that maximizes compute and memory resources, causing memory bandwidth underutilization in the prefill phase and compute underutilization in the decode phase. Such underutilization directly translates into increased serving costs. This paper proposes SPAD (Specialized Prefill and Decode hardware), adopting a less-is-more methodology to design specialized chips tailored to the distinct characteristics of prefill and decode phases. The proposed Prefill Chips have larger systolic arrays and use cost-effective GDDR memory, whereas the proposed Decode Chips retain high memory bandwidth but reduce compute capacity. Compared to modeled H100s, simulations show that the proposed Prefill Chips deliver 8% higher prefill performance on average at 52% lower hardware cost, while the proposed Decode Chips achieve 97% of the decode performance with 28% lower TDP. End-to-end simulations on production traces show that SPAD reduces hardware cost by 19%-41% and TDP by 2%-17% compared to modeled baseline clusters while offering the same performance. Even when models and workloads change, SPAD can reallocate either type of chip to run either phase and still achieve 11%-43% lower hardware costs, demonstrating the longevity of the SPAD design.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 25 of its 64 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

25 of 64 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  9. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
13 more
  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  4. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  5. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  6. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  7. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  8. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  9. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  10. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  11. 2024Mirage: A Multi-Level Superoptimizer for Tensor Programscited by 41
  12. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 25
  13. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokenscited by 5
  2. 2026Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Modelscited by 3
  3. 2026Towards Green AI: Decoding the Energy of LLM Inference in Software Developmentcited by 2
  4. 2026Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decodingcited by 0
  5. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  6. 2026NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  2. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  3. 2023Efficient LLM inference solution on Intel GPU
  4. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  5. 2025Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs
  6. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs
  7. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  8. 2025T-MAN: Enabling End-to-End Low-Bit LLM Inference on NPUs via Unified Table Lookup