# Short-Term Memory Convolutions

arXiv: https://arxiv.org/abs/2302.04331 · HTML: https://mlsanity.com/paper/2302.04331/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 8 Feb 2023.

## Builds on (most cited first, 17 of 17)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [Locality defeats the curse of dimensionality in convolutional teacher-student scenarios](https://mlsanity.com/paper/2106.08619/) (2021)
- [Multi-scale Multi-band DenseNets for Audio Source Separation](https://mlsanity.com/paper/1706.09588/) (2017)
- [Deeper Image Quality Transfer: Training Low-Memory Neural Networks for 3D Images](https://mlsanity.com/paper/1808.05577/) (2018)
- [Exploring Tradeoffs in Models for Low-latency Speech Enhancement](https://mlsanity.com/paper/1811.07030/) (2018)
- [Efficient Low-Latency Speech Enhancement with Mobile Audio Streaming Networks](https://mlsanity.com/paper/2008.07244/) (2020)
- [Near Memory Acceleration on High Resolution Radio Astronomy Imaging](https://mlsanity.com/paper/2005.04098/) (2020)

## Built on by (most cited first, 3 of 3)

- [EmMixformer: Mix transformer for eye movement recognition](https://mlsanity.com/paper/2401.04956/) (2024)
- [SOI: Scaling Down Computational Complexity by Estimating Partial States of the Model](https://mlsanity.com/paper/2410.03813/) (2024)
- [Sub-Model Short-Term Memory Convolutions for Keyword Spotting Systems on Device](https://mlsanity.com/paper/2609.35005/) (2026)

## Abstract (arXiv)

The real-time processing of time series signals is a critical issue for many real-life applications. The idea of real-time processing is especially important in audio domain as the human perception of sound is sensitive to any kind of disturbance in perceived signals, especially the lag between auditory and visual modalities. The rise of deep learning (DL) models complicated the landscape of signal processing. Although they often have superior quality compared to standard DSP methods, this advantage is diminished by higher latency. In this work we propose novel method for minimization of inference time latency and memory consumption, called Short-Term Memory Convolution (STMC) and its transposed counterpart. The main advantage of STMC is the low latency comparable to long short-term memory (LSTM) networks. Furthermore, the training of STMC-based models is faster and more stable as the method is based solely on convolutional neural networks (CNNs). In this study we demonstrate an application of this solution to a U-Net model for a speech separation task and GhostNet model in acoustic scene classification (ASC) task. In case of speech separation we achieved a 5-fold reduction in inference time and a 2-fold reduction in latency without affecting the output quality. The inference time for ASC task was up to 4 times faster while preserving the original accuracy.