# Hadamard Domain Training with Integers for Class Incremental Quantized Learning

arXiv: https://arxiv.org/abs/2310.03675 · HTML: https://mlsanity.com/paper/2310.03675/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 72 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 30 of 38)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks](https://mlsanity.com/paper/1903.08066/) (2019)
- [Optimal Continual Learning has Perfect Memory and is NP-hard](https://mlsanity.com/paper/2006.05188/) (2020)
- [Training Transformers with 4-bit Integers](https://mlsanity.com/paper/2306.11987/) (2023)
- [SparCL: Sparse Continual Learning on the Edge](https://mlsanity.com/paper/2209.09476/) (2022)
- [Adjusting Decision Boundary for Class Imbalanced Learning](https://mlsanity.com/paper/1912.01857/) (2019)
- [La-MAML: Look-ahead Meta Learning for Continual Learning](https://mlsanity.com/paper/2007.13904/) (2020)
- [A Statistical Framework for Low-bitwidth Training of Deep Neural Networks](https://mlsanity.com/paper/2010.14298/) (2020)
- [Online Continual Learning for Embedded Devices](https://mlsanity.com/paper/2203.10681/) (2022)
- [Stratified Transfer Learning for Cross-domain Activity Recognition](https://mlsanity.com/paper/1801.00820/) (2017)
- [A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays](https://mlsanity.com/paper/2110.10486/) (2021)
- [Continual Learning via Bit-Level Information Preserving](https://mlsanity.com/paper/2105.04444/) (2021)
- [FP8 versus INT8 for efficient deep learning inference](https://mlsanity.com/paper/2303.17951/) (2023)
- [Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats](https://mlsanity.com/paper/2112.10769/) (2021)
- [SIESTA: Efficient Online Continual Learning with Sleep](https://mlsanity.com/paper/2303.10725/) (2023)
- [How Efficient Are Today's Continual Learning Algorithms?](https://mlsanity.com/paper/2303.18171/) (2023)

## Built on by (most cited first, 3 of 3)

- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [HOT: Hadamard-based Optimized Training](https://mlsanity.com/paper/2503.21261/) (2025)
- [AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation](https://mlsanity.com/paper/2604.02525/) (2026)

## Abstract (arXiv)

Continual learning is a desirable feature in many modern machine learning applications, which allows in-field adaptation and updating, ranging from accommodating distribution shift, to fine-tuning, and to learning new tasks. For applications with privacy and low latency requirements, the compute and memory demands imposed by continual learning can be cost-prohibitive for resource-constraint edge platforms. Reducing computational precision through fully quantized training (FQT) simultaneously reduces memory footprint and increases compute efficiency for both training and inference. However, aggressive quantization especially integer FQT typically degrades model accuracy to unacceptable levels. In this paper, we propose a technique that leverages inexpensive Hadamard transforms to enable low-precision training with only integer matrix multiplications. We further determine which tensors need stochastic rounding and propose tiled matrix multiplication to enable low-bit width accumulators. We demonstrate the effectiveness of our technique on several human activity recognition datasets and CIFAR100 in a class incremental learning setting. We achieve less than 0.5% and 3% accuracy degradation while we quantize all matrix multiplications inputs down to 4-bits with 8-bit accumulators.