# Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking

arXiv: https://arxiv.org/abs/1804.06826 · HTML: https://mlsanity.com/paper/1804.06826/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2018.

## Built on by (most cited first, 30 of 35)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [ThinK: Thinner Key Cache by Query-Driven Pruning](https://mlsanity.com/paper/2407.21018/) (2024)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [MaxK-GNN: Extremely Fast GPU Kernel Design for Accelerating Graph Neural Networks Training](https://mlsanity.com/paper/2312.08656/) (2023)
- [Accel-GCN: High-Performance GPU Accelerator Design for Graph Convolution Networks](https://mlsanity.com/paper/2308.11825/) (2023)
- [Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models](https://mlsanity.com/paper/2203.12758/) (2022)
- [Ditto: Accelerating Diffusion Model via Temporal Value Similarity](https://mlsanity.com/paper/2501.11211/) (2025)
- [Post-Training Sparsity-Aware Quantization](https://mlsanity.com/paper/2105.11010/) (2021)
- [Efficient Quantized Sparse Matrix Operations on Tensor Cores](https://mlsanity.com/paper/2209.06979/) (2022)
- [FastSeq: Make Sequence Generation Faster](https://mlsanity.com/paper/2106.04718/) (2021)
- [APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Cores](https://mlsanity.com/paper/2106.12169/) (2021)
- [Collage: Light-Weight Low-Precision Strategy for LLM Training](https://mlsanity.com/paper/2405.03637/) (2024)
- [GPU-Based Homotopy Continuation for Minimal Problems in Computer Vision](https://mlsanity.com/paper/2112.03444/) (2021)
- [Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs](https://mlsanity.com/paper/2403.08845/) (2024)
- [Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning](https://mlsanity.com/paper/2110.10548/) (2021)
- [Insight Gained from Migrating a Machine Learning Model to Intelligence Processing Units](https://mlsanity.com/paper/2404.10730/) (2024)
- [E2Former-V2: On-the-Fly Equivariant Attention with Linear Activation Memory](https://mlsanity.com/paper/2601.16622/) (2026)
- [AdaSplash-2: Faster Differentiable Sparse Attention](https://mlsanity.com/paper/2604.15180/) (2026)
- [Optimized Speculative Sampling for GPU Hardware Accelerators](https://mlsanity.com/paper/2406.11016/) (2024)
- [Debunking the CUDA Myth Towards GPU-based AI Systems](https://mlsanity.com/paper/2501.00210/) (2024)
- [Rethinking LLM Advancement: Compute-Dependent and Independent Paths to Progress](https://mlsanity.com/paper/2505.04075/) (2025)
- [KernelSight-LM: A Kernel-Level LLM Inference Simulator](https://mlsanity.com/paper/2606.28565/) (2026)
- [A Study on the Intersection of GPU Utilization and CNN Inference](https://mlsanity.com/paper/2212.07936/) (2022)
- [The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks](https://mlsanity.com/paper/2310.00496/) (2023)
- [Memory-Efficient Acceleration of Block Low-Rank Foundation Models on Resource Constrained GPUs](https://mlsanity.com/paper/2512.20861/) (2025)
- [M100: An Orchestrated Dataflow Architecture Powering General AI Computing](https://mlsanity.com/paper/2604.17862/) (2026)
- [On the Noise Stability and Robustness of Adversarially Trained Networks on NVM Crossbars](https://mlsanity.com/paper/2109.09060/) (2021)
- [Accelerating Fully Connected Neural Network on Optical Network-on-Chip (ONoC)](https://mlsanity.com/paper/2109.14878/) (2021)

## Abstract (arXiv)

Every year, novel NVIDIA GPU designs are introduced. This rapid architectural and technological progression, coupled with a reluctance by manufacturers to disclose low-level details, makes it difficult for even the most proficient GPU software designers to remain up-to-date with the technological advances at a microarchitectural level. To address this dearth of public, microarchitectural-level information on the novel NVIDIA GPUs, independent researchers have resorted to microbenchmarks-based dissection and discovery. This has led to a prolific line of publications that shed light on instruction encoding, and memory hierarchy's geometry and features at each level. Namely, research that describes the performance and behavior of the Kepler, Maxwell and Pascal architectures. In this technical report, we continue this line of research by presenting the microarchitectural details of the NVIDIA Volta architecture, discovered through microbenchmarks and instruction set disassembly. Additionally, we compare quantitatively our Volta findings against its predecessors, Kepler, Maxwell and Pascal.