mlsanityData to 7 Oct 2026

In-Datacenter Performance Analysis of a Tensor Processing Unit

arXiv 1704.04760 ↗First posted 16 Apr 2017cs.AR cs.LG cs.NEcited by 610

Not on the map. Why?

Abstract (arXiv)

Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU)---deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs (caches, out-of-order execution, multithreading, multiprocessing, prefetching, ...) that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power. We compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU, which are contemporaries deployed in the same datacenters. Our workload, written in the high-level TensorFlow framework, uses production NN applications (MLPs, CNNs, and LSTMs) that represent 95% of our datacenters' NN inference demand. Despite low utilization for some applications, the TPU is on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU.

Key facts. Cited by 610 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 1; 2021: 184; 2022: 118; 2023: 105; 2024: 76; 2025: 73; 2026: 52.

610 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  4. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  7. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  8. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  9. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  10. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  11. 2022Matryoshka Representation Learningcited by 268
  12. 2021Combinatorial optimization and reasoning with graph neural networkscited by 267
38 more
  1. 2021Urban Radiance Fieldscited by 263
  2. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  3. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  4. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  5. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  6. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  7. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  8. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  9. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  10. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  11. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  12. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  13. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  14. 2023Large Language Models in Law: A Surveycited by 128
  15. 2023End-to-End Speech Recognition: A Surveycited by 127
  16. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  17. 2021Large-Scale Differentially Private BERTcited by 117
  18. 2024Let Your Graph Do the Talking: Encoding Structured Data for LLMscited by 112
  19. 2021Dive into Deep Learningcited by 111
  20. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  21. 2021Efficient training of physics-informed neural networks via importance samplingcited by 109
  22. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  23. 2021Colorization Transformercited by 89
  24. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  25. 2021Proof-of-Learning: Definitions and Practicecited by 81
  26. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  27. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  28. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  29. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  30. 2021TT-Rec: Tensor Train Compression for Deep Learning Recommendation Modelscited by 58
  31. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 58
  32. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  33. 2021A Survey of Machine Learning for Computer Architecture and Systemscited by 53
  34. 2021Tensor Methods in Computer Vision and Deep Learningcited by 52
  35. 2021Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Frameworkcited by 52
  36. 2022Description-Driven Task-Oriented Dialog Modelingcited by 50
  37. 2023Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipelinecited by 50
  38. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49

50 most cited of 610 All 610 (JSON)