# WaferLLM: Large Language Model Inference at Wafer Scale

arXiv: https://arxiv.org/abs/2502.04563 · HTML: https://mlsanity.com/paper/2502.04563/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 16 of its 52 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism](https://mlsanity.com/paper/2404.09526/) (2024)
- [FlashDecoding++: Faster Large Language Model Inference on GPUs](https://mlsanity.com/paper/2311.01282/) (2023)
- [TENET: A Framework for Modeling Tensor Dataflow Based on Relation-centric Notation](https://mlsanity.com/paper/2105.01892/) (2021)
- [DISTAL: The Distributed Tensor Algebra Compiler](https://mlsanity.com/paper/2203.08069/) (2022)
- [Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10](https://mlsanity.com/paper/2408.04808/) (2024)
- [Near-Optimal Wafer-Scale Reduce](https://mlsanity.com/paper/2404.15888/) (2024)

## Built on by (most cited first, 13 of 13)

- [Agentic AI Home Energy Management System: A Large Language Model Framework for Residential Load Scheduling](https://mlsanity.com/paper/2510.26603/) (2025)
- [ContextPilot: Fast Long-Context Inference via Context Reuse](https://mlsanity.com/paper/2511.03475/) (2025)
- [Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference](https://mlsanity.com/paper/2510.05497/) (2025)
- [A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search](https://mlsanity.com/paper/2507.00004/) (2025)
- [A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA](https://mlsanity.com/paper/2603.05931/) (2026)
- [Agentic Witnessing: Pragmatic and Scalable TEE-Enabled Privacy-Preserving Auditing](https://mlsanity.com/paper/2604.24203/) (2026)
- [SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer](https://mlsanity.com/paper/2607.28150/) (2026)
- [Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM](https://mlsanity.com/paper/2510.05632/) (2025)
- [Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling](https://mlsanity.com/paper/2603.27624/) (2026)
- [KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving](https://mlsanity.com/paper/2605.09735/) (2026)
- [MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelining Framework for Prefill-Only LLM Inference](https://mlsanity.com/paper/2606.22968/) (2026)
- [Cache-Resident LLM Inference in GB-Scale Last-Level Caches](https://mlsanity.com/paper/2606.25353/) (2026)
- [Vortex: Bridging Extreme Compression and Efficient LLM Inference](https://mlsanity.com/paper/2609.12208/) (2026)

## Abstract (arXiv)

Emerging AI accelerators increasingly adopt wafer-scale manufacturing technologies, integrating hundreds of thousands of AI cores in a mesh architecture with large distributed on-chip memory (tens of GB in total) and ultra-high on-chip memory bandwidth (tens of PB/s). However, current LLM inference systems, optimized for shared memory architectures like GPUs, fail to exploit these accelerators fully. We introduce WaferLLM, the first wafer-scale LLM inference system. WaferLLM is guided by a novel PLMR model (pronounced as "Plummer") that captures the unique hardware characteristics of wafer-scale architectures. Leveraging this model, WaferLLM pioneers wafer-scale LLM parallelism, optimizing the utilization of hundreds of thousands of on-chip cores. It also introduces MeshGEMM and MeshGEMV, the first GEMM and GEMV implementations designed to scale effectively on wafer-scale accelerators. Evaluations show that WaferLLM achieves up to 200× higher accelerator utilization than state-of-the-art methods. Leveraging a wafer-scale accelerator (Cerebras WSE2), WaferLLM delivers GEMV operations 606× faster and 16× more energy-efficient than on an NVIDIA A100 GPU. For full LLM inference, WaferLLM achieves 10-20× speedups over A100 GPU clusters running SGLang and vLLM. These advantages are expected to grow as wafer-scale AI models, software, and hardware continue to mature. WaferLLM is open-sourced at https://github.com/MeshInfra/WaferLLM.