# Implicit Feature Decoupling with Depthwise Quantization

arXiv: https://arxiv.org/abs/2203.08080 · HTML: https://mlsanity.com/paper/2203.08080/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 26 of 26)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Robust Classification with Convolutional Prototype Learning](https://mlsanity.com/paper/1805.03438/) (2018)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [And the Bit Goes Down: Revisiting the Quantization of Neural Networks](https://mlsanity.com/paper/1907.05686/) (2019)
- [Theory and Experiments on Vector Quantized Autoencoders](https://mlsanity.com/paper/1805.11063/) (2018)
- [Z-Forcing: Training Stochastic Recurrent Networks](https://mlsanity.com/paper/1711.05411/) (2017)
- [Estimating Mutual Information for Discrete-Continuous Mixtures](https://mlsanity.com/paper/1709.06212/) (2017)
- [Rethinking Depthwise Separable Convolutions: How Intra-Kernel Correlations Lead to Improved MobileNets](https://mlsanity.com/paper/2003.13549/) (2020)
- [Network Decoupling: From Regular to Depthwise Separable Convolutions](https://mlsanity.com/paper/1808.05517/) (2018)
- [Fast Wavenet Generation Algorithm](https://mlsanity.com/paper/1611.09482/) (2016)
- [Learning Product Codebooks using Vector Quantized Autoencoders for Image Retrieval](https://mlsanity.com/paper/1807.04629/) (2018)
- [Variational Information Bottleneck on Vector Quantized Autoencoders](https://mlsanity.com/paper/1808.01048/) (2018)
- [Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval](https://mlsanity.com/paper/2103.12328/) (2021)
- [Convolutional Prototype Learning for Zero-Shot Recognition](https://mlsanity.com/paper/1910.09728/) (2019)

## Abstract (arXiv)

Quantization has been applied to multiple domains in Deep Neural Networks (DNNs). We propose Depthwise Quantization (DQ) where textitquantization is applied to a decomposed sub-tensor along the textitfeature axis of weak statistical dependence. The feature decomposition leads to an exponential increase in textitrepresentation capacity with a linear increase in memory and parameter cost. In addition, DQ can be directly applied to existing encoder-decoder frameworks without modification of the DNN architecture. We use DQ in the context of Hierarchical Auto-Encoder and train end-to-end on an image feature representation. We provide an analysis on cross-correlation between spatial and channel features and we propose a decomposition of the image feature representation along the channel axis. The improved performance of the depthwise operator is due to the increased representation capacity from implicit feature decoupling. We evaluate DQ on the likelihood estimation task, where it outperforms the previous state-of-the-art on CIFAR-10, ImageNet-32 and ImageNet-64. We progressively train with increasing image size a single hierarchical model that uses 69% less parameters and has a faster convergence than the previous works.