HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens, while the remaining experts are lightly loaded. On 3.5D multi-chiplet systems, this skew not only causes compute imbalance but also amplifies pressure on communication, memory bandwidth, I/O, and execution queues. Therefore, the core problem is not simply to reduce token movement, but to dynamically place and reuse hot expert replicas across different memory tiers. This paper proposes HCRMap, a hot expert residency mapping framework for pressure-aware expert replica management in 3.5D MoE inference. Based on expert hotness, weight loading cost, migration overhead, and runtime resource pressure, HCRMap dynamically determines which experts should be promoted, retained, demoted, or evicted. It then maps routed token groups to suitable resident replicas, thereby jointly mitigating communication, memory, and queue bottlenecks. Experimental results show that HCRMap reduces end-to-end latency by 43.6% and 43.0% over Hydra in the prefill and decode stages, respectively; by 34.5% and 33.1% over MoEntwine; and by 46.7% and 46.0% over PIMoE.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 10 of its 31 references that are in the map. First posted on arXiv 13 Jul 2026.
Builds on
10 of 31 references.
Closest ancestors · 7
Same direction, newest first
- 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing2 Jun
- 2026Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures7 Mar
- 2026PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching31 Jan
- 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving12 Jan
- 2025MoEntwine: Unleashing the Potential of Wafer-scale Chips for Large-scale Expert Parallel Inference29 Oct
- 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference29 Sep
- 2024MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models29 May
Other references · 3
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2025LaMoSys3.5D: Enabling 3.5D-IC-Based Large Language Model Inference Serving Systems via Hardware/Software Co-Designcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling
- 2026ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts
- 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
- 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
- 2026TAOT: Topology-Aware Optimal Transport for Dynamic Expert Replica Placement in MoE Training
- 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
- 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
- 2025A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration