# Benchmarking and Dissecting the Nvidia Hopper GPU Architecture

arXiv: https://arxiv.org/abs/2402.13499 · HTML: https://mlsanity.com/paper/2402.13499/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2024.

## Built on by (most cited first, 24 of 24)

- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [A Survey of Small Language Models](https://mlsanity.com/paper/2410.20011/) (2024)
- [Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey](https://mlsanity.com/paper/2405.00314/) (2024)
- [LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind](https://mlsanity.com/paper/2508.15601/) (2025)
- [BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache](https://mlsanity.com/paper/2503.18773/) (2025)
- [ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression](https://mlsanity.com/paper/2603.17435/) (2026)
- [FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness](https://mlsanity.com/paper/2412.03317/) (2024)
- [MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity](https://mlsanity.com/paper/2511.13061/) (2025)
- [Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores](https://mlsanity.com/paper/2409.17870/) (2024)
- [FlashKAT: Understanding and Addressing Performance Bottlenecks in the Kolmogorov-Arnold Transformer](https://mlsanity.com/paper/2505.13813/) (2025)
- [BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers](https://mlsanity.com/paper/2507.03117/) (2025)
- [LC-Opt: Benchmarking Reinforcement Learning and Agentic AI for End-to-End Liquid Cooling Optimization in Data Centers](https://mlsanity.com/paper/2511.00116/) (2025)
- [DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model](https://mlsanity.com/paper/2505.19349/) (2025)
- [Energy and polarization based on-line interference mitigation in radio interferometry](https://mlsanity.com/paper/2412.14775/) (2024)
- [APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration](https://mlsanity.com/paper/2508.19087/) (2025)
- [APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing](https://mlsanity.com/paper/2606.08761/) (2026)
- [Dissecting GPU Utilization for LLM Inference on Nvidia Hopper](https://mlsanity.com/paper/2609.12923/) (2026)
- [Pushing the Performance Envelope of DNN-based Recommendation Systems Inference on GPUs](https://mlsanity.com/paper/2410.22249/) (2024)
- [TriADA: Massively Parallel Trilinear Matrix-by-Tensor Multiply-Add Algorithm and Device Architecture for the Acceleration of 3D Discrete Transformations](https://mlsanity.com/paper/2506.22818/) (2025)
- [TriGen: NPU Architecture for End-to-End Acceleration of Large Language Models based on SW-HW Co-Design](https://mlsanity.com/paper/2602.12962/) (2026)
- [A Unified Memory Perspective for Probabilistic Trustworthy AI](https://mlsanity.com/paper/2603.25692/) (2026)
- [Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariants](https://mlsanity.com/paper/2604.18616/) (2026)
- [FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessment](https://mlsanity.com/paper/2608.14550/) (2026)
- [Maia 200: A Software Defined Dataflow System for Large-scale AI Acceleration](https://mlsanity.com/paper/2608.24664/) (2026)

## Abstract (arXiv)

Graphics processing units (GPUs) are continually evolving to cater to the computational demands of contemporary general-purpose workloads, particularly those driven by artificial intelligence (AI) utilizing deep learning techniques. A substantial body of studies have been dedicated to dissecting the microarchitectural metrics characterizing diverse GPU generations, which helps researchers understand the hardware details and leverage them to optimize the GPU programs. However, the latest Hopper GPUs present a set of novel attributes, including new tensor cores supporting FP8, DPX, and distributed shared memory. Their details still remain mysterious in terms of performance and operational characteristics. In this research, we propose an extensive benchmarking study focused on the Hopper GPU. The objective is to unveil its microarchitectural intricacies through an examination of the new instruction-set architecture (ISA) of Nvidia GPUs and the utilization of new CUDA APIs. Our approach involves two main aspects. Firstly, we conduct conventional latency and throughput comparison benchmarks across the three most recent GPU architectures, namely Hopper, Ada, and Ampere. Secondly, we delve into a comprehensive discussion and benchmarking of the latest Hopper features, encompassing the Hopper DPX dynamic programming (DP) instruction set, distributed shared memory, and the availability of FP8 tensor cores. The microbenchmarking results we present offer a deeper understanding of the novel GPU AI function units and programming features introduced by the Hopper architecture. This newfound understanding is expected to greatly facilitate software optimization and modeling efforts for GPU architectures. To the best of our knowledge, this study makes the first attempt to demystify the tensor core performance and programming instruction sets unique to Hopper GPUs.