# RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference

arXiv: https://arxiv.org/abs/2002.11921 · HTML: https://mlsanity.com/paper/2002.11921/

## Key facts

- Cited by 21 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 27 Feb 2020.

## Built on by (most cited first, 21 of 21)

- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning](https://mlsanity.com/paper/2110.15352/) (2021)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [SIRNN: A Math Library for Secure RNN Inference](https://mlsanity.com/paper/2105.04236/) (2021)
- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [Network Augmentation for Tiny Deep Learning](https://mlsanity.com/paper/2110.08890/) (2021)
- [P2M: A Processing-in-Pixel-in-Memory Paradigm for Resource-Constrained TinyML Applications](https://mlsanity.com/paper/2203.04737/) (2022)
- [Implicit Bias of Linear RNNs](https://mlsanity.com/paper/2101.07833/) (2021)
- [Going Deeper Into Face Detection: A Survey](https://mlsanity.com/paper/2103.14983/) (2021)
- [MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory](https://mlsanity.com/paper/2310.16898/) (2023)
- [Self-Attentive Pooling for Efficient Deep Learning](https://mlsanity.com/paper/2209.07659/) (2022)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Dynamic Iterative Refinement for Efficient 3D Hand Pose Estimation](https://mlsanity.com/paper/2111.06500/) (2021)
- [MinUn: Accurate ML Inference on Microcontrollers](https://mlsanity.com/paper/2210.16556/) (2022)
- [Pex: Memory-efficient Microcontroller Deep Learning through Partial Execution](https://mlsanity.com/paper/2211.17246/) (2022)
- [Improving the Energy Efficiency and Robustness of tinyML Computer Vision using Log-Gradient Input Images](https://mlsanity.com/paper/2203.02571/) (2022)
- [Enabling ISP-less Low-Power Computer Vision](https://mlsanity.com/paper/2210.05451/) (2022)
- [ConformalLayers: A non-linear sequential neural network with associative layers](https://mlsanity.com/paper/2110.12108/) (2021)
- [Multi-output Headed Ensembles for Product Item Classification](https://mlsanity.com/paper/2307.15858/) (2023)
- [Value-Driven Mixed-Precision Quantization for Patch-Based Inference on Microcontrollers](https://mlsanity.com/paper/2401.13714/) (2024)

## Abstract (arXiv)

Standard Convolutional Neural Networks (CNNs) designed for computer vision tasks tend to have large intermediate activation maps. These require large working memory and are thus unsuitable for deployment on resource-constrained devices typically used for inference on the edge. Aggressively downsampling the images via pooling or strided convolutions can address the problem but leads to a significant decrease in accuracy due to gross aggregation of the feature map by standard pooling operators. In this paper, we introduce RNNPool, a novel pooling operator based on Recurrent Neural Networks (RNNs), that efficiently aggregates features over large patches of an image and rapidly downsamples activation maps. Empirical evaluation indicates that an RNNPool layer can effectively replace multiple blocks in a variety of architectures such as MobileNets, DenseNet when applied to standard vision tasks like image classification and face detection. That is, RNNPool can significantly decrease computational complexity and peak memory usage for inference while retaining comparable accuracy. We use RNNPool with the standard S3FD architecture to construct a face detection method that achieves state-of-the-art MAP for tiny ARM Cortex-M4 class microcontrollers with under 256 KB of RAM. Code is released at https://github.com/Microsoft/EdgeML.