# Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systems

arXiv: https://arxiv.org/abs/2208.11035 · HTML: https://mlsanity.com/paper/2208.11035/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2022.

## Built on by (most cited first, 9 of 9)

- [Cascade Speculative Drafting for Even Faster LLM Inference](https://mlsanity.com/paper/2312.11462/) (2023)
- [MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism](https://mlsanity.com/paper/2504.02263/) (2025)
- [POLCA: Power Oversubscription in LLM Cloud Providers](https://mlsanity.com/paper/2308.12908/) (2023)
- [What Limits Agentic Systems Efficiency?](https://mlsanity.com/paper/2510.16276/) (2025)
- [AI Benchmark Democratization and Carpentry](https://mlsanity.com/paper/2512.11588/) (2025)
- [ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving](https://mlsanity.com/paper/2606.00735/) (2026)
- [Balanced and Elastic End-to-end Training of Dynamic LLMs](https://mlsanity.com/paper/2505.14864/) (2025)
- [Detecting Anomalies in Machine Learning Infrastructure via Hardware Telemetry](https://mlsanity.com/paper/2510.26008/) (2025)
- [GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems](https://mlsanity.com/paper/2605.19945/) (2026)

## Abstract (arXiv)

Scientists are increasingly exploring and utilizing the massive parallelism of general-purpose accelerators such as GPUs for scientific breakthroughs. As a result, datacenters, hyperscalers, national computing centers, and supercomputers have procured hardware to support this evolving application paradigm. These systems contain hundreds to tens of thousands of accelerators, enabling peta- and exa-scale levels of compute for scientific workloads. Recent work demonstrated that power management (PM) can impact application performance in CPU-based HPC systems, even when machines have the same architecture and SKU (stock keeping unit). This variation occurs due to manufacturing variability and the chip's PM. However, while modern HPC systems widely employ accelerators such as GPUs, it is unclear how much this variability affects applications. Accordingly, we seek to characterize the extent of variation due to GPU PM in modern HPC and supercomputing systems. We study a variety of applications that stress different GPU components on five large-scale computing centers with modern GPUs: Oak Ridge's Summit, Sandia's Vortex, TACC's Frontera and Longhorn, and Livermore's Corona. These clusters use a variety of cooling methods and GPU vendors. In total, we collect over 18,800 hours of data across more than 90% of the GPUs in these clusters. Regardless of the application, cluster, GPU vendor, and cooling method, our results show significant variation: 8% (max 22%) average performance variation even though the GPU architecture and vendor SKU are identical within each cluster, with outliers up to 1.5X slower than the median GPU. These results highlight the difficulty in efficiently using existing GPU clusters for modern HPC and scientific workloads, and the need to embrace variability in future accelerator-based systems.