# AI Accelerators for Large Language Model Inference: Architecture Analysis and Scaling Strategies

arXiv: https://arxiv.org/abs/2506.00008 · HTML: https://mlsanity.com/paper/2506.00008/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 45 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 1 of 1)

- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)

## Built on by (most cited first, 1 of 1)

- [AI Compute Architecture and Evolution Trends](https://mlsanity.com/paper/2508.21394/) (2025)

## Abstract (arXiv)

The rapid growth of large-language models (LLMs) is driving a new wave of specialized hardware for inference. This paper presents the first workload-centric, cross-architectural performance study of commercial AI accelerators, spanning GPU-based chips, hybrid packages, and wafer-scale engines. We compare memory hierarchies, compute fabrics, and on-chip interconnects, and observe up to 3.7x performance variation across architectures as batch size and sequence length change. Four scaling techniques for trillion-parameter models are examined; expert parallelism offers an 8.4x parameter-to-compute advantage but incurs 2.1x higher latency variance than tensor parallelism. These findings provide quantitative guidance for matching workloads to accelerators and reveal architectural gaps that next-generation designs must address.