Towards Building Private LLMs: Exploring Multi-Node Expert Parallelism on Apple Silicon for Mixture-of-Experts Large Language Model
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI) with significant advancements such as OpenAI's ChatGPT, Meta's Llama, and Databricks' DBRX. This paper addresses the cost and scalability challenges encountered when constructing private LLM systems for personal or small group services, as aimed by Apple Intelligence. A Mac Studio cluster with Apple's M2 Ultra chips is established as a cost-efficient solution to host and accelerate the pretrained DBRX model with the Mixture-of-Experts (MoE) architecture. Our performance analysis reveal that parallel execution of the model's experts across two to four machine nodes significantly reduces inference time. We find that computation time for the experts is comparable to the communication time for exchanging their outputs, emphasizing the importance of network latency over bandwidth. We also observe significant management overhead due to Apple software stack's memory management logic. Based on these findings, we develop optimization schemes to eliminate the memory management overhead. As a result, the Mac Studio cluster is 1.15 times more cost-efficient than the state-of-the-art AI supercomputer with NVIDIA H100 GPUs. In addition, we construct a performance model to estimate system performance under varying configurations, and the model provides valuable insights for designing private LLM systems.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 6 of its 20 references that are in the map. First posted on arXiv 30 Jun 2025.
Builds on
6 of 20 references.
References · 6
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
- 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
- 2025Viability and Performance of a Private LLM Server for SMBs: A Benchmark Analysis of Qwen3-30B on Consumer-Grade Hardware
- 2026MoEless: Efficient MoE LLM Serving via Serverless Computing
- 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training
- 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
- 2025Cache Management for Mixture-of-Experts LLMs -- extended version
- 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
- 2026Federation of Experts: Communication Efficient Distributed Inference for Large Language Models