# HeTraX: Energy Efficient 3D Heterogeneous Manycore Architecture for Transformer Acceleration

arXiv: https://arxiv.org/abs/2408.03397 · HTML: https://mlsanity.com/paper/2408.03397/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 17 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 5 of 5)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [DNN+NeuroSim V2.0: An End-to-End Benchmarking Framework for Compute-in-Memory Accelerators for On-chip Training](https://mlsanity.com/paper/2003.06471/) (2020)
- [X-Former: In-Memory Acceleration of Transformers](https://mlsanity.com/paper/2303.07470/) (2023)
- [Learning-based Application-Agnostic 3D NoC Design for Heterogeneous Manycore Systems](https://mlsanity.com/paper/1810.08869/) (2018)

## Built on by (most cited first, 1 of 1)

- [ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts](https://mlsanity.com/paper/2607.17074/) (2026)

## Abstract (arXiv)

Transformers have revolutionized deep learning and generative modeling to enable unprecedented advancements in natural language processing tasks and beyond. However, designing hardware accelerators for executing transformer models is challenging due to the wide variety of computing kernels involved in the transformer architecture. Existing accelerators are either inadequate to accelerate end-to-end transformer models or suffer notable thermal limitations. In this paper, we propose the design of a three-dimensional heterogeneous architecture referred to as HeTraX specifically optimized to accelerate end-to-end transformer models. HeTraX employs hardware resources aligned with the computational kernels of transformers and optimizes both performance and energy. Experimental results show that HeTraX outperforms existing state-of-the-art by up to 5.6x in speedup and improves EDP by 14.5x while ensuring thermally feasibility.