mlsanityData to 7 Oct 2026

Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures

arXiv 2603.07006 ↗First posted 7 Mar 2026cs.ARcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) architecture offers enhanced efficiency for Large Language Models (LLMs) with modularized computation, yet its inherent sparsity poses significant hardware deployment challenges, including memory locality issues, communication overhead, and inefficient computing resource utilization. Inspired by the modular organization of the human brain, we propose Mozart, a novel algorithm-hardware co-design framework tailored for efficient training of MoE-based LLMs on 3.5D wafer-scale chiplet architectures. On the algorithm side, Mozart exploits the inherent modularity of chiplets and introduces: (1) an expert allocation strategy that enables efficient on-package all-to-all communication, and (2) a fine-grained scheduling mechanism that improves communication-computation overlap through streaming tokens and experts. On the architecture side, Mozart adaptively co-locates heterogeneous modules on specialized chiplets with a 2.5D NoP-Tree topology and hierarchical memory structure. Evaluation across three popular MoE models demonstrates significant efficiency gains, enabling more effective parallelization and resource utilization for large-scale modularized MoE-LLMs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 18 of its 45 references that are in the map. First posted on arXiv 7 Mar 2026.

Builds on

18 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design2 Apr
  2. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun

Other references · 15

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2024Mixtral of Expertscited by 1,785
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  10. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  11. 20242 OLMo 2 Furiouscited by 302
  12. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
3 more
  1. 2021Brain-inspired computing: We need a master plancited by 51
  2. 2024Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLMcited by 17
  3. 2021Impact of On-Chip Interconnect on In-Memory Acceleration of Deep Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts
  2. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling
  3. 2025Towards Building Private LLMs: Exploring Multi-Node Expert Parallelism on Apple Silicon for Mixture-of-Experts Large Language Model
  4. 2025A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration
  5. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
  6. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
  7. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training
  8. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production