# Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM

arXiv: https://arxiv.org/abs/2510.05632 · HTML: https://mlsanity.com/paper/2510.05632/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 27 of its 81 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 27 of 27)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills](https://mlsanity.com/paper/2308.16369/) (2023)
- [Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration](https://mlsanity.com/paper/2406.01014/) (2024)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration](https://mlsanity.com/paper/1911.09925/) (2019)
- [NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing](https://mlsanity.com/paper/2403.00579/) (2024)
- [SpArch: Efficient Architecture for Sparse Matrix Multiplication](https://mlsanity.com/paper/2002.08947/) (2020)
- [ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale](https://mlsanity.com/paper/2303.14006/) (2023)
- [LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale](https://mlsanity.com/paper/2408.05499/) (2024)
- [Sparseloop: An Analytical Approach To Sparse Tensor Accelerator Modeling](https://mlsanity.com/paper/2205.05826/) (2022)
- [APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving](https://mlsanity.com/paper/2411.17651/) (2024)
- [WaferLLM: Large Language Model Inference at Wafer Scale](https://mlsanity.com/paper/2502.04563/) (2025)
- [CiMLoop: A Flexible, Accurate, and Fast Compute-In-Memory Modeling Tool](https://mlsanity.com/paper/2405.07259/) (2024)
- [ONNXim: A Fast, Cycle-level Multi-core NPU Simulator](https://mlsanity.com/paper/2406.08051/) (2024)
- [Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10](https://mlsanity.com/paper/2408.04808/) (2024)
- [System and Design Technology Co-optimization of SOT-MRAM for High-Performance AI Accelerator Memory System](https://mlsanity.com/paper/2303.12310/) (2023)
- [RapidChiplet: A Toolchain for Rapid Design Space Exploration of Chiplet Architectures](https://mlsanity.com/paper/2311.06081/) (2023)
- [DCRA: A Distributed Chiplet-based Reconfigurable Architecture for Irregular Applications](https://mlsanity.com/paper/2311.15443/) (2023)

## Abstract (arXiv)

With the widespread adoption of Large Language Models (LLMs), the demand for high-performance LLM inference services continues to grow. Multi-core AI accelerators, such as Groq, Graphcore IPU, and Cerebras WSE, provide promising platforms for LLM serving, but their distributed memory systems require careful coordination between hardware configuration and serving policies. Otherwise, mismatched tensor partitioning, data placement, and memory management can substantially underutilize compute and communication resources. To address these challenges, we present WaferAI-SIM, a multi-level simulation framework that combines transaction-level simulation with an analytical performance model. WaferAI-SIM enables simulator-driven co-design of LLM serving strategies and multi-core accelerator architectures, targeting the early design stage in which emerging platforms are not yet broadly available for empirical serving studies. It captures how LLM serving policies interact with compute-core count, memory hierarchy, and interconnect topology, enabling architecture-aware exploration beyond GPU-centric assumptions. We evaluate representative LLMs across a range of chip configurations and serving scenarios. Across the evaluation, WaferAI-SIM reports 1.32×--6.03× latency improvements, where the lower endpoint comes from ring-based placement at TP=16 over the placement baselines, and the upper endpoint comes from K-dimension TP over MN-dimension TP for Qwen34B at TP=4 with sequence length 256. For LLM serving, our findings provide guidance for co-designing hardware architectures and serving strategies for multi-core AI accelerators across diverse LLM workloads.