# Dissecting the Graphcore IPU Architecture via Microbenchmarking

arXiv: https://arxiv.org/abs/1912.03413 · HTML: https://mlsanity.com/paper/1912.03413/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2019.

## Built on by (most cited first, 30 of 47)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [AEGNN: Asynchronous Event-based Graph Neural Networks](https://mlsanity.com/paper/2203.17149/) (2022)
- [Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets](https://mlsanity.com/paper/2310.04292/) (2023)
- [Performance of a Geometric Deep Learning Pipeline for HL-LHC Particle Tracking](https://mlsanity.com/paper/2103.06995/) (2021)
- [Efficient recurrent architectures through activity sparsity and sparse back-propagation through time](https://mlsanity.com/paper/2206.06178/) (2022)
- [Flashlight: Enabling Innovation in Tools for Machine Learning](https://mlsanity.com/paper/2201.12465/) (2022)
- [A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms](https://mlsanity.com/paper/2306.15552/) (2023)
- [Training and inference of large language models using 8-bit floating point](https://mlsanity.com/paper/2309.17224/) (2023)
- [Dynamic Sparsity Is Channel-Level Sparsity Learner](https://mlsanity.com/paper/2305.19454/) (2023)
- [Unit Scaling: Out-of-the-Box Low-Precision Training](https://mlsanity.com/paper/2303.11257/) (2023)
- [Graph Neural Networks in Particle Physics: Implementations, Innovations, and Challenges](https://mlsanity.com/paper/2203.12852/) (2022)
- [Multi-Horizon Forecasting for Limit Order Books: Novel Deep Learning Approaches and Hardware Acceleration using Intelligent Processing Units](https://mlsanity.com/paper/2105.10430/) (2021)
- [Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques](https://mlsanity.com/paper/2112.15131/) (2021)
- [MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation](https://mlsanity.com/paper/2302.10872/) (2023)
- [Design Principles for Lifelong Learning AI Accelerators](https://mlsanity.com/paper/2310.04467/) (2023)
- [Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance](https://mlsanity.com/paper/2107.02027/) (2021)
- [Training Large Language Models Efficiently with Sparsity and Dataflow](https://mlsanity.com/paper/2304.05511/) (2023)
- [Photonic Reconfigurable Accelerators for Efficient Inference of CNNs with Mixed-Sized Tensors](https://mlsanity.com/paper/2207.05278/) (2022)
- [Evaluating Emerging AI/ML Accelerators: IPU, RDU, and NVIDIA/AMD GPUs](https://mlsanity.com/paper/2311.04417/) (2023)
- [Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10](https://mlsanity.com/paper/2408.04808/) (2024)
- [Adaptive Block Floating-Point for Analog Deep Learning Hardware](https://mlsanity.com/paper/2205.06287/) (2022)
- [Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models](https://mlsanity.com/paper/2311.04386/) (2023)
- [Extreme Acceleration of Graph Neural Network-based Prediction Models for Quantum Chemistry](https://mlsanity.com/paper/2211.13853/) (2022)
- [Efficient Real Time Recurrent Learning through combined activity and parameter sparsity](https://mlsanity.com/paper/2303.05641/) (2023)
- [Optimal Gradient Checkpointing for Sparse and Recurrent Architectures using Off-Chip Memory](https://mlsanity.com/paper/2412.11810/) (2024)
- [GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures](https://mlsanity.com/paper/2106.05822/) (2021)
- [Physics Community Needs, Tools, and Resources for Machine Learning](https://mlsanity.com/paper/2203.16255/) (2022)
- [Performance and Power: Systematic Evaluation of AI Workloads on Accelerators with CARAML](https://mlsanity.com/paper/2409.12994/) (2024)
- [GPU Domain Specialization via Composable On-Package Architecture](https://mlsanity.com/paper/2104.02188/) (2021)

## Abstract (arXiv)

This report focuses on the architecture and performance of the Intelligence Processing Unit (IPU), a novel, massively parallel platform recently introduced by Graphcore and aimed at Artificial Intelligence/Machine Learning (AI/ML) workloads. We dissect the IPU's performance behavior using microbenchmarks that we crafted for the purpose. We study the IPU's memory organization and performance. We study the latency and bandwidth that the on-chip and off-chip interconnects offer, both in point-to-point transfers and in a spectrum of collective operations, under diverse loads. We evaluate the IPU's compute power over matrix multiplication, convolution, and AI/ML primitives. We discuss actual performance in comparison with its theoretical limits. Our findings reveal how the IPU's architectural design affects its performance. Moreover, they offer simple mental models to predict an application's performance on the IPU, on the basis of the computation and communication steps it involves. This report is the natural extension to a novel architecture of a continuing effort of ours that focuses on the microbenchmark-based discovery of massively parallel architectures.