# NVIDIA Tensor Core Programmability, Performance & Precision

arXiv: https://arxiv.org/abs/1803.04014 · HTML: https://mlsanity.com/paper/1803.04014/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2018.

## Built on by (most cited first, 30 of 79)

- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)
- [LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models](https://mlsanity.com/paper/2206.09557/) (2022)
- [Proof-of-Learning: Definitions and Practice](https://mlsanity.com/paper/2103.05633/) (2021)
- [MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization](https://mlsanity.com/paper/2405.17873/) (2024)
- [Mirage: A Multi-Level Superoptimizer for Tensor Programs](https://mlsanity.com/paper/2405.05751/) (2024)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [Fast Neural Representations for Direct Volume Rendering](https://mlsanity.com/paper/2112.01579/) (2021)
- [Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization](https://mlsanity.com/paper/2403.12422/) (2024)
- [A Machine Learning-oriented Survey on Tiny Machine Learning](https://mlsanity.com/paper/2309.11932/) (2023)
- [Flashlight: Enabling Innovation in Tools for Machine Learning](https://mlsanity.com/paper/2201.12465/) (2022)
- [Quantization-Aware Imitation-Learning for Resource-Efficient Robotic Control](https://mlsanity.com/paper/2412.01034/) (2024)
- [PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation](https://mlsanity.com/paper/2106.14681/) (2021)
- [TPU-KNN: K Nearest Neighbor Search at Peak FLOP/s](https://mlsanity.com/paper/2206.14286/) (2022)
- [Post-Training Sparsity-Aware Quantization](https://mlsanity.com/paper/2105.11010/) (2021)
- [Allo: A Programming Model for Composable Accelerator Design](https://mlsanity.com/paper/2404.04815/) (2024)
- [Clairvoyant Prefetching for Distributed Machine Learning I/O](https://mlsanity.com/paper/2101.08734/) (2021)
- [PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation](https://mlsanity.com/paper/2407.11798/) (2024)
- [MARCA: Mamba Accelerator with ReConfigurable Architecture](https://mlsanity.com/paper/2409.11440/) (2024)
- [SECDA: Efficient Hardware/Software Co-Design of FPGA-based DNN Accelerators for Edge Inference](https://mlsanity.com/paper/2110.00478/) (2021)
- [DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction](https://mlsanity.com/paper/2203.11014/) (2022)
- [Saliency-Aware Quantized Imitation Learning for Efficient Robotic Control](https://mlsanity.com/paper/2505.15304/) (2025)
- [Three-dimensional microstructure generation using generative adversarial neural networks in the context of continuum micromechanics](https://mlsanity.com/paper/2206.01693/) (2022)
- [Design Space Exploration of Low-Bit Quantized Neural Networks for Visual Place Recognition](https://mlsanity.com/paper/2312.09028/) (2023)
- [QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2502.12346/) (2025)
- [Reference-based Defect Detection Network](https://mlsanity.com/paper/2108.04456/) (2021)
- [Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelis](https://mlsanity.com/paper/2406.18820/) (2024)
- [Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models](https://mlsanity.com/paper/2102.02344/) (2021)
- [Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale](https://mlsanity.com/paper/2105.12676/) (2021)
- [Ascend HiFloat8 Format for Deep Learning](https://mlsanity.com/paper/2409.16626/) (2024)
- [Quant 4.0: Engineering Quantitative Investment with Automated, Explainable and Knowledge-driven Artificial Intelligence](https://mlsanity.com/paper/2301.04020/) (2022)

## Abstract (arXiv)

The NVIDIA Volta GPU microarchitecture introduces a specialized unit, called "Tensor Core" that performs one matrix-multiply-and-accumulate on 4x4 matrices per clock cycle. The NVIDIA Tesla V100 accelerator, featuring the Volta microarchitecture, provides 640 Tensor Cores with a theoretical peak performance of 125 Tflops/s in mixed precision. In this paper, we investigate current approaches to program NVIDIA Tensor Cores, their performances and the precision loss due to computation in mixed precision. Currently, NVIDIA provides three different ways of programming matrix-multiply-and-accumulate on Tensor Cores: the CUDA Warp Matrix Multiply Accumulate (WMMA) API, CUTLASS, a templated library based on WMMA, and cuBLAS GEMM. After experimenting with different approaches, we found that NVIDIA Tensor Cores can deliver up to 83 Tflops/s in mixed precision on a Tesla V100 GPU, seven and three times the performance in single and half precision respectively. A WMMA implementation of batched GEMM reaches a performance of 4 Tflops/s. While precision loss due to matrix multiplication with half precision input might be critical in many HPC applications, it can be considerably reduced at the cost of increased computation. Our results indicate that HPC applications using matrix multiplications can strongly benefit from using of NVIDIA Tensor Cores.