# Analyzing GPU Tensor Core Potential for Fast Reductions

arXiv: https://arxiv.org/abs/1903.03640 · HTML: https://mlsanity.com/paper/1903.03640/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2019.

## Built on by (most cited first, 1 of 1)

- [Learning from distinctive candidates to optimize reduced-precision convolution program on tensor cores](https://mlsanity.com/paper/2202.06819/) (2022)

## Abstract (arXiv)

The Nvidia GPU architecture has introduced new computing elements such as the textittensor cores, which are special processing units dedicated to perform fast matrix-multiply-accumulate (MMA) operations and accelerate textitDeep Learning applications. In this work we present the idea of using tensor cores for a different purpose such as the parallel arithmetic reduction problem, and propose a new GPU tensor-core based algorithm as well as analyze its potential performance benefits in comparison to a traditional GPU-based one. The proposed method, encodes the reduction of n numbers as a set of m× m MMA tensor-core operations (for Nvidia's Volta architecture m=16) and takes advantage from the fact that each MMA operation takes just one GPU cycle. When analyzing the cost under a simplified GPU computing model, the result is that the new algorithm manages to reduce a problem of n numbers in T(n) = 5logm2(n) steps with a speedup of S = frac45log2(m2).