# Memory Efficient Adaptive Attention For Multiple Domain Learning

arXiv: https://arxiv.org/abs/2110.10969 · HTML: https://mlsanity.com/paper/2110.10969/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 21 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Superposition of many models into one](https://mlsanity.com/paper/1902.05522/) (2019)
- [Depthwise Convolution is All You Need for Learning Multiple Visual Domains](https://mlsanity.com/paper/1902.00927/) (2019)
- [Knowledge transfer in deep block-modular neural networks](https://mlsanity.com/paper/1908.08017/) (2019)
- [Feature Partitioning for Efficient Multi-Task Architectures](https://mlsanity.com/paper/1908.04339/) (2019)
- [NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks](https://mlsanity.com/paper/2006.12813/) (2020)
- [Not all domains are equally complex: Adaptive Multi-Domain Learning](https://mlsanity.com/paper/2003.11504/) (2020)
- [Multi-path Neural Networks for On-device Multi-domain Visual Classification](https://mlsanity.com/paper/2010.04904/) (2020)

## Abstract (arXiv)

Training CNNs from scratch on new domains typically demands large numbers of labeled images and computations, which is not suitable for low-power hardware. One way to reduce these requirements is to modularize the CNN architecture and freeze the weights of the heavier modules, that is, the lower layers after pre-training. Recent studies have proposed alternative modular architectures and schemes that lead to a reduction in the number of trainable parameters needed to match the accuracy of fully fine-tuned CNNs on new domains. Our work suggests that a further reduction in the number of trainable parameters by an order of magnitude is possible. Furthermore, we propose that new modularization techniques for multiple domain learning should also be compared on other realistic metrics, such as the number of interconnections needed between the fixed and trainable modules, the number of training samples needed, the order of computations required and the robustness to partial mislabeling of the training data. On all of these criteria, the proposed architecture demonstrates advantages over or matches the current state-of-the-art.