mlsanityData to 7 Oct 2026

HadaCore: Tensor Core Accelerated Hadamard Transform Kernel

arXiv 2412.08832 ↗First posted 12 Dec 2024cs.DC cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

We present HadaCore, a modified Fast Walsh-Hadamard Transform (FWHT) algorithm optimized for the Tensor Cores present in modern GPU hardware. HadaCore follows the recursive structure of the original FWHT algorithm, achieving the same asymptotic runtime complexity but leveraging a hardware-aware work decomposition that benefits from Tensor Core acceleration. This reduces bottlenecks from compute and data exchange. On Nvidia A100 and H100 GPUs, HadaCore achieves speedups of 1.1-1.4x and 1.0-1.3x, with a peak gain of 3.5x and 3.6x respectively, when compared to the existing state-of-the-art implementation of the original algorithm. We also show that when using FP16 or BF16, our implementation is numerically accurate, enabling comparable accuracy on MMLU benchmarks when used in an end-to-end Llama3 inference run with quantized (FP8) attention.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 5 of its 8 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

5 of 8 references.

References · 5

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  3. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  4. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  5. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 16
  2. 2025Efficient Data Selection at Scale via Influence Distillationcited by 14
  3. 2026Provable Quantization with Randomized Hadamard Transformcited by 6
  4. 2025RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithmcited by 5
  5. 2025ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformerscited by 5
  6. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  7. 2025Improving Quantization with Post-Training Model Expansioncited by 4
  8. 2026LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Losscited by 1
  9. 2026FoldQuantVLA: Native Low-Bit Quantization of Vision-Language-Action Models via Consistent Foldingcited by 0