mlsanityData to 14 Oct 2026

HOT: Hadamard-based Optimized Training

arXiv 2503.21261 ↗First posted 27 Mar 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

It has become increasingly important to optimize backpropagation to reduce memory usage and computational overhead. Achieving this goal is highly challenging, as multiple objectives must be considered jointly while maintaining training quality. In this paper, we focus on matrix multiplication, which accounts for the largest portion of training costs, and analyze its backpropagation in detail to identify lightweight techniques that offer the best benefits. Based on this analysis, we introduce a novel method, Hadamard-based Optimized Training (HOT). In this approach, we apply Hadamard-based optimizations, such as Hadamard quantization and Hadamard low-rank approximation, selectively and with awareness of the suitability of each optimization for different backward paths. Additionally, we introduce two enhancements: activation buffer compression and layer-wise quantizer selection. Our extensive analysis shows that HOT achieves up to 75% memory savings and a 2.6 times acceleration on real GPUs, with negligible accuracy loss compared to FP32 precision.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

34 of 47 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  10. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
22 more
  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  5. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 309
  6. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  7. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  8. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  9. 2022FP8 Formats for Deep Learningcited by 196
  10. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  11. 2023Training Transformers with 4-bit Integerscited by 60
  12. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  13. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  14. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  15. 2018NICE: Noise Injection and Clamping Estimation for Neural Network Quantizationcited by 24
  16. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  17. 2024ResMaster: Mastering High-Resolution Image Generation via Structural and Fine-Grained Guidancecited by 22
  18. 2020NITI: Training Integer Neural Networks Using Integer-only Arithmeticcited by 20
  19. 2022Is Integer Arithmetic Enough for Deep Learning Training?cited by 12
  20. 2023Efficient Low-rank Backpropagation for Vision Transformer Adaptationcited by 10
  21. 2023Hadamard Domain Training with Integers for Class Incremental Quantized Learningcited by 3
  22. 2024UniMem: Towards a Unified View of Long-Context Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 9
  2. 2026AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024HLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantization
  2. 2024HadaCore: Tensor Core Accelerated Hadamard Transform Kernel
  3. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs
  4. 2026Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
  5. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  6. 2026FlashOptim: Optimizers for Memory-Efficient Training
  7. 2026Mixed-Precision Training and Compilation for RRAM-based Computing-in-Memory Accelerators
  8. 2025DPQ-HD: Post-Training Compression for Ultra-Low Power Hyperdimensional Computing