# Summarizing CPU and GPU Design Trends with Product Data

arXiv: https://arxiv.org/abs/1911.11313 · HTML: https://mlsanity.com/paper/1911.11313/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2019.

## Built on by (most cited first, 10 of 10)

- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Reversible Vision Transformers](https://mlsanity.com/paper/2302.04869/) (2023)
- [NetKet 3: Machine Learning Toolbox for Many-Body Quantum Systems](https://mlsanity.com/paper/2112.10526/) (2021)
- [NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention](https://mlsanity.com/paper/2403.01273/) (2024)
- [Beyond Efficiency: Scaling AI Sustainably](https://mlsanity.com/paper/2406.05303/) (2024)
- [A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities](https://mlsanity.com/paper/2111.14247/) (2021)
- [EnergyVis: Interactively Tracking and Exploring Energy Consumption for ML Models](https://mlsanity.com/paper/2103.16435/) (2021)
- [ISimDL: Importance Sampling-Driven Acceleration of Fault Injection Simulations for Evaluating the Robustness of Deep Learning](https://mlsanity.com/paper/2303.08035/) (2023)
- [Democratizing AI: A Comparative Study in Deep Learning Efficiency and Future Trends in Computational Processing](https://mlsanity.com/paper/2603.20920/) (2026)

## Abstract (arXiv)

Moore's Law and Dennard Scaling have guided the semiconductor industry for the past few decades. Recently, both laws have faced validity challenges as transistor sizes approach the practical limits of physics. We are interested in testing the validity of these laws and reflect on the reasons responsible. In this work, we collect data of more than 4000 publicly-available CPU and GPU products. We find that transistor scaling remains critical in keeping the laws valid. However, architectural solutions have become increasingly important and will play a larger role in the future. We observe that GPUs consistently deliver higher performance than CPUs. GPU performance continues to rise because of increases in GPU frequency, improvements in the thermal design power (TDP), and growth in die size. But we also see the ratio of GPU to CPU performance moving closer to parity, thanks to new SIMD extensions on CPUs and increased CPU core counts.