# Learnable Mixed-precision and Dimension Reduction Co-design for Low-storage Activation

arXiv: https://arxiv.org/abs/2207.07931 · HTML: https://mlsanity.com/paper/2207.07931/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 16 Jul 2022.

## Builds on (most cited first, 14 of 14)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [APQ: Joint Search for Network Architecture, Pruning and Quantization Policy](https://mlsanity.com/paper/2006.08509/) (2020)
- [Rethinking Differentiable Search for Mixed-Precision Neural Networks](https://mlsanity.com/paper/2004.05795/) (2020)
- [Towards Effective Low-bitwidth Convolutional Neural Networks](https://mlsanity.com/paper/1711.00205/) (2017)
- [FracBits: Mixed Precision Quantization via Fractional Bit-Widths](https://mlsanity.com/paper/2007.02017/) (2020)
- [Efficient and Accurate Approximations of Nonlinear Convolutional Networks](https://mlsanity.com/paper/1411.4229/) (2014)
- [ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks](https://mlsanity.com/paper/1811.01704/) (2018)
- [Feature Map Transform Coding for Energy-Efficient CNN Inference](https://mlsanity.com/paper/1905.10830/) (2019)
- [Structured Binary Neural Networks for Image Recognition](https://mlsanity.com/paper/1909.09934/) (2019)
- [Compression-aware Projection with Greedy Dimension Reduction for Convolutional Neural Network Activations](https://mlsanity.com/paper/2110.08828/) (2021)

## Built on by (most cited first, 2 of 2)

- [MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2401.14895/) (2024)
- [TSPTQ-ViT: Two-scaled post-training quantization for vision transformer](https://mlsanity.com/paper/2305.12901/) (2023)

## Abstract (arXiv)

Recently, deep convolutional neural networks (CNNs) have achieved many eye-catching results. However, deploying CNNs on resource-constrained edge devices is constrained by limited memory bandwidth for transmitting large intermediated data during inference, i.e., activation. Existing research utilizes mixed-precision and dimension reduction to reduce computational complexity but pays less attention to its application for activation compression. To further exploit the redundancy in activation, we propose a learnable mixed-precision and dimension reduction co-design system, which separates channels into groups and allocates specific compression policies according to their importance. In addition, the proposed dynamic searching technique enlarges search space and finds out the optimal bit-width allocation automatically. Our experimental results show that the proposed methods improve 3.54%/1.27% in accuracy and save 0.18/2.02 bits per value over existing mixed-precision methods on ResNet18 and MobileNetv2, respectively.