# GPU Domain Specialization via Composable On-Package Architecture

arXiv: https://arxiv.org/abs/2104.02188 · HTML: https://mlsanity.com/paper/2104.02188/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 79 references that are in the map.
- First posted on arXiv 5 Apr 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Deep Learning Face Representation by Joint Identification-Verification](https://mlsanity.com/paper/1406.4773/) (2014)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [Dissecting the Graphcore IPU Architecture via Microbenchmarking](https://mlsanity.com/paper/1912.03413/) (2019)
- [PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units](https://mlsanity.com/paper/1909.04548/) (2019)
- [LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference](https://mlsanity.com/paper/2010.13103/) (2020)
- [Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training](https://mlsanity.com/paper/1907.13257/) (2019)

## Built on by (most cited first, 1 of 1)

- [Optical Transformers](https://mlsanity.com/paper/2302.10360/) (2023)

## Abstract (arXiv)

As GPUs scale their low precision matrix math throughput to boost deep learning (DL) performance, they upset the balance between math throughput and memory system capabilities. We demonstrate that converged GPU design trying to address diverging architectural requirements between FP32 (or larger) based HPC and FP16 (or smaller) based DL workloads results in sub-optimal configuration for either of the application domains. We argue that a Composable On-PAckage GPU (COPAGPU) architecture to provide domain-specialized GPU products is the most practical solution to these diverging requirements. A COPA-GPU leverages multi-chip-module disaggregation to support maximal design reuse, along with memory system specialization per application domain. We show how a COPA-GPU enables DL-specialized products by modular augmentation of the baseline GPU architecture with up to 4x higher off-die bandwidth, 32x larger on-package cache, 2.3x higher DRAM bandwidth and capacity, while conveniently supporting scaled-down HPC-oriented designs. This work explores the microarchitectural design necessary to enable composable GPUs and evaluates the benefits composability can provide to HPC, DL training, and DL inference. We show that when compared to a converged GPU design, a DL-optimized COPA-GPU featuring a combination of 16x larger cache capacity and 1.6x higher DRAM bandwidth scales per-GPU training and inference performance by 31% and 35% respectively and reduces the number of GPU instances by 50% in scale-out training scenarios.