# ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts

arXiv: https://arxiv.org/abs/2607.17074 · HTML: https://mlsanity.com/paper/2607.17074/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 19 Jul 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 10 of 10)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Accelerating Distributed MoE Training and Inference with Lina](https://mlsanity.com/paper/2210.17223/) (2022)
- [DNN+NeuroSim V2.0: An End-to-End Benchmarking Framework for Compute-in-Memory Accelerators for On-chip Training](https://mlsanity.com/paper/2003.06471/) (2020)
- [X-Former: In-Memory Acceleration of Transformers](https://mlsanity.com/paper/2303.07470/) (2023)
- [Learning-based Application-Agnostic 3D NoC Design for Heterogeneous Manycore Systems](https://mlsanity.com/paper/1810.08869/) (2018)
- [MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training](https://mlsanity.com/paper/2501.03905/) (2025)
- [CHIME: Chiplet-based Heterogeneous Near-Memory Acceleration for Edge Multimodal LLM Inference](https://mlsanity.com/paper/2601.19908/) (2025)
- [Atleus: Accelerating Transformers on the Edge Enabled by 3D Heterogeneous Manycore Architectures](https://mlsanity.com/paper/2501.09588/) (2025)
- [A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration](https://mlsanity.com/paper/2507.19142/) (2025)
- [HeTraX: Energy Efficient 3D Heterogeneous Manycore Architecture for Transformer Acceleration](https://mlsanity.com/paper/2408.03397/) (2024)

## Abstract (arXiv)

Mixture of Experts (MoE) architectures have emerged as a dominant paradigm for scaling Large Language Models (LLMs). However, MoE inference on conventional hardware is constrained by three fundamental bottlenecks. These encompass the massive memory bandwidth required to fetch non-contiguous expert weights, the non-deterministic scatter-gather traffic generated by input-dependent token routing, and the tail-latency dependency imposed by synchronous expert output aggregation. To address these challenges, we propose ThAME, a three-dimensional (3D) heterogeneous multi-chiplet architecture for MoE inference. ThAME employs Ferroelectric Field-Effect Transistor (FeFET)-based non-volatile and DRAM-based volatile memory chiplets with a co-designed compute mapping strategy that aligns the distinct computational profiles of attention mechanisms and expert routing. Furthermore, we design a specialized Network-on-Chip communication backbone optimized to mitigate the bottlenecks associated with non-deterministic token routing traffic across the combinatorial space of input-dependent MoE traffic patterns. Experimental results demonstrate that ThAME outperforms state-of-the-art counterparts by up to 15.7x in terms of speedup and improves energy efficiency by up to 9.8x.