mlsanityData to 13 Oct 2026

WaferLLM: Large Language Model Inference at Wafer Scale

arXiv 2502.04563 ↗First posted 6 Feb 2025cs.LG cs.AI cs.ARcited by 13

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Emerging AI accelerators increasingly adopt wafer-scale manufacturing technologies, integrating hundreds of thousands of AI cores in a mesh architecture with large distributed on-chip memory (tens of GB in total) and ultra-high on-chip memory bandwidth (tens of PB/s). However, current LLM inference systems, optimized for shared memory architectures like GPUs, fail to exploit these accelerators fully. We introduce WaferLLM, the first wafer-scale LLM inference system. WaferLLM is guided by a novel PLMR model (pronounced as "Plummer") that captures the unique hardware characteristics of wafer-scale architectures. Leveraging this model, WaferLLM pioneers wafer-scale LLM parallelism, optimizing the utilization of hundreds of thousands of on-chip cores. It also introduces MeshGEMM and MeshGEMV, the first GEMM and GEMV implementations designed to scale effectively on wafer-scale accelerators. Evaluations show that WaferLLM achieves up to 200× higher accelerator utilization than state-of-the-art methods. Leveraging a wafer-scale accelerator (Cerebras WSE2), WaferLLM delivers GEMV operations 606× faster and 16× more energy-efficient than on an NVIDIA A100 GPU. For full LLM inference, WaferLLM achieves 10-20× speedups over A100 GPU clusters running SGLang and vLLM. These advantages are expected to grow as wafer-scale AI models, software, and hardware continue to mature. WaferLLM is open-sourced at https://github.com/MeshInfra/WaferLLM.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 52 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

16 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  3. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov

Other references · 13

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,117
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 626
  9. 2022Efficiently Scaling Transformer Inferencecited by 532
  10. 2021TENET: A Framework for Modeling Tensor Dataflow Based on Relation-centric Notationcited by 9
  11. 2022DISTAL: The Distributed Tensor Algebra Compilercited by 6
  12. 2024Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10cited by 5
1 more
  1. 2024Near-Optimal Wafer-Scale Reducecited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025Agentic AI Home Energy Management System: A Large Language Model Framework for Residential Load Schedulingcited by 8
  2. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  3. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  4. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  5. 2026A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGAcited by 3
  6. 2026Agentic Witnessing: Pragmatic and Scalable TEE-Enabled Privacy-Preserving Auditingcited by 3
  7. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  8. 2025Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIMcited by 0
  9. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  10. 2026KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Servingcited by 0
  11. 2026MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelining Framework for Prefill-Only LLM Inferencecited by 0
  12. 2026Cache-Resident LLM Inference in GB-Scale Last-Level Cachescited by 0
1 more
  1. 2026Vortex: Bridging Extreme Compression and Efficient LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall ↗
  2. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  3. 2026BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal
  4. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs
  5. 2025AxLLM: accelerator architecture for large language models with computation reuse capability
  6. 2026BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators
  7. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  8. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design