# Serving Large Language Models on Huawei CloudMatrix384

arXiv: https://arxiv.org/abs/2506.12708 · HTML: https://mlsanity.com/paper/2506.12708/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- First posted on arXiv 15 Jun 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Built on by (most cited first, 24 of 24)

- [Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding](https://mlsanity.com/paper/2507.19427/) (2025)
- [Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management](https://mlsanity.com/paper/2511.20172/) (2025)
- [TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving](https://mlsanity.com/paper/2508.17219/) (2025)
- [xLLM Technical Report](https://mlsanity.com/paper/2510.14686/) (2025)
- [W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs](https://mlsanity.com/paper/2601.16536/) (2026)
- [ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs](https://mlsanity.com/paper/2604.03298/) (2026)
- [P/D-Device: Disaggregated Large Language Model between Cloud and Devices](https://mlsanity.com/paper/2508.09035/) (2025)
- [HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission](https://mlsanity.com/paper/2510.19470/) (2025)
- [RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference](https://mlsanity.com/paper/2601.01712/) (2026)
- [Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads](https://mlsanity.com/paper/2601.21351/) (2026)
- [Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs](https://mlsanity.com/paper/2604.18788/) (2026)
- [Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving](https://mlsanity.com/paper/2606.09061/) (2026)
- [Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design](https://mlsanity.com/paper/2606.10493/) (2026)
- [AMLA: MUL by ADD in FlashAttention Rescaling](https://mlsanity.com/paper/2509.25224/) (2025)
- [A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPU](https://mlsanity.com/paper/2602.17693/) (2026)
- [SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD](https://mlsanity.com/paper/2607.20145/) (2026)
- [TyphoonMLA: A Mixed Naive-Absorb MLA Kernel For Shared Prefix](https://mlsanity.com/paper/2509.21081/) (2025)
- [TASP: Topology-aware Sequence Parallelism](https://mlsanity.com/paper/2509.26541/) (2025)
- [From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models](https://mlsanity.com/paper/2511.02248/) (2025)
- [OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency](https://mlsanity.com/paper/2511.22481/) (2025)
- [HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures](https://mlsanity.com/paper/2602.00748/) (2026)
- [ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill](https://mlsanity.com/paper/2606.22541/) (2026)
- [OpScale: Operator-level Provisioning and Autoscaling for LLM Serving](https://mlsanity.com/paper/2608.13499/) (2026)
- [HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processing](https://mlsanity.com/paper/2609.08682/) (2026)

## Abstract (arXiv)

The rapid evolution of large language models (LLMs), driven by growing parameter scales, adoption of mixture-of-experts (MoE) architectures, and expanding context lengths, imposes unprecedented demands on AI infrastructure. Traditional AI clusters face limitations in compute intensity, memory bandwidth, inter-chip communication, and latency, compounded by variable workloads and strict service-level objectives. Addressing these issues requires fundamentally redesigned hardware-software integration. This paper introduces Huawei CloudMatrix, a next-generation AI datacenter architecture, realized in the production-grade CloudMatrix384 supernode. It integrates 384 Ascend 910 NPUs and 192 Kunpeng CPUs interconnected via an ultra-high-bandwidth Unified Bus (UB) network, enabling direct all-to-all communication and dynamic pooling of resources. These features optimize performance for communication-intensive operations, such as large-scale MoE expert parallelism and distributed key-value cache access. To fully leverage CloudMatrix384, we propose CloudMatrix-Infer, an advanced LLM serving solution incorporating three core innovations: a peer-to-peer serving architecture that independently scales prefill, decode, and caching; a large-scale expert parallelism strategy supporting EP320 via efficient UB-based token dispatch; and hardware-aware optimizations including specialized operators, microbatch-based pipelining, and INT8 quantization. Evaluation with the DeepSeek-R1 model shows CloudMatrix-Infer achieves state-of-the-art efficiency: prefill throughput of 6,688 tokens/s per NPU and decode throughput of 1,943 tokens/s per NPU (<50 ms TPOT). It effectively balances throughput and latency, sustaining 538 tokens/s per NPU even under stringent 15 ms latency constraints, while INT8 quantization maintains model accuracy across benchmarks.