mlsanityData to 7 Oct 2026

AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation

arXiv 2604.02525 ↗First posted 2 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Hadamard transforms have become a key tool for stabilizing low-precision training, but existing methods apply them uniformly across tensors and computation paths. We show that this one-size-fits-all strategy is inherently limited: Hadamard smoothing reduces quantization error only when its direction is properly aligned with the operand's outlier structure. Through a systematic study of weights, activations, and gradients in LLM training, we identify three stable outlier patterns, Row-wise, Column-wise, and None, and show that each outlier pattern pair in matrix multiplication requires a distinct transform or outlier-handling strategy. We propose AdaHOP, Adaptive Hadamard transform with Outlier-Pattern-aware strategy, which applies Inner Hadamard Transform (IHT) when inner-dimension mixing properly suppresses the operands' outliers, and selectively applies Outlier Extraction (OE) that extracts dominant outlier rows or columns into a high-precision path when it does not. With fused, hardware-aware Triton kernels, AdaHOP enables training from scratch at MXFP4 precision with BF16-level quality, while achieving up to 3.6X memory compression, 1.46X end-to-end training speedup over BF16.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 46 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

41 of 46 references.

References · 41

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2024Mixtral of Expertscited by 1,785
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
29 more
  1. 2017Mixed Precision Trainingcited by 1,135
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  3. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  6. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  7. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  8. 2015Deep Learning with Limited Numerical Precisioncited by 368
  9. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  10. 2022FP8 Formats for Deep Learningcited by 187
  11. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  12. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  13. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 80
  14. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  15. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 62
  16. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  17. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 37
  18. 2025Training LLMs with MXFP4cited by 36
  19. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  20. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  21. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 30
  22. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  23. 2020NITI: Training Integer Neural Networks Using Integer-only Arithmeticcited by 18
  24. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 16
  25. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 14
  26. 2023Efficient Low-rank Backpropagation for Vision Transformer Adaptationcited by 10
  27. 2025Instella: Fully Open Language Models with Stellar Performancecited by 4
  28. 2023Hadamard Domain Training with Integers for Class Incremental Quantized Learningcited by 3
  29. 2025HOT: Hadamard-based Optimized Trainingcited by 2

Built on by

Not cited yet.