# HadaCore: Tensor Core Accelerated Hadamard Transform Kernel

arXiv: https://arxiv.org/abs/2412.08832 · HTML: https://mlsanity.com/paper/2412.08832/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 8 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 5 of 5)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)

## Built on by (most cited first, 12 of 12)

- [Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models](https://mlsanity.com/paper/2505.17761/) (2025)
- [Efficient Data Selection at Scale via Influence Distillation](https://mlsanity.com/paper/2505.19051/) (2025)
- [Provable Quantization with Randomized Hadamard Transform](https://mlsanity.com/paper/2605.13810/) (2026)
- [ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformers](https://mlsanity.com/paper/2512.03673/) (2025)
- [Improving Quantization with Post-Training Model Expansion](https://mlsanity.com/paper/2503.17513/) (2025)
- [RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithm](https://mlsanity.com/paper/2504.03717/) (2025)
- [OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond](https://mlsanity.com/paper/2605.19660/) (2026)
- [LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss](https://mlsanity.com/paper/2605.08755/) (2026)
- [Transforms for LLM Quantization: The Great Inversion and Format Co-Design](https://mlsanity.com/paper/2608.25188/) (2026)
- [Memristive-Friendly Hadamard Reservoir Computing: Structured, Multiplier-Free Recurrences at Scale](https://mlsanity.com/paper/2608.28295/) (2026)
- [FoldQuantVLA: Native Low-Bit Quantization of Vision-Language-Action Models via Consistent Folding](https://mlsanity.com/paper/2609.24433/) (2026)
- [Fast Quantized Johnson-Lindenstrauss Transforms](https://mlsanity.com/paper/2610.16289/) (2026)

## Abstract (arXiv)

We present HadaCore, a modified Fast Walsh-Hadamard Transform (FWHT) algorithm optimized for the Tensor Cores present in modern GPU hardware. HadaCore follows the recursive structure of the original FWHT algorithm, achieving the same asymptotic runtime complexity but leveraging a hardware-aware work decomposition that benefits from Tensor Core acceleration. This reduces bottlenecks from compute and data exchange. On Nvidia A100 and H100 GPUs, HadaCore achieves speedups of 1.1-1.4x and 1.0-1.3x, with a peak gain of 3.5x and 3.6x respectively, when compared to the existing state-of-the-art implementation of the original algorithm. We also show that when using FP16 or BF16, our implementation is numerically accurate, enabling comparable accuracy on MMLU benchmarks when used in an end-to-end Llama3 inference run with quantized (FP8) attention.