# Sequence Modeling with Multiresolution Convolutional Memory

arXiv: https://arxiv.org/abs/2305.01638 · HTML: https://mlsanity.com/paper/2305.01638/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 63 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Sparse Sinkhorn Attention](https://mlsanity.com/paper/2002.11296/) (2020)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)

## Built on by (most cited first, 19 of 19)

- [Test-time regression: a unifying framework for designing sequence models with associative memory](https://mlsanity.com/paper/2501.12352/) (2025)
- [Parallelizing non-linear sequential models over the sequence length](https://mlsanity.com/paper/2309.12252/) (2023)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [Spectral State Space Models](https://mlsanity.com/paper/2312.06837/) (2023)
- [Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective](https://mlsanity.com/paper/2402.11463/) (2024)
- [Hybrid2 Neural ODE Causal Modeling and an Application to Glycemic Response](https://mlsanity.com/paper/2402.17233/) (2024)
- [FutureFill: Fast Generation from Convolutional Sequence Models](https://mlsanity.com/paper/2410.03766/) (2024)
- [Simplified Sparse Attention via Gist Tokens](https://mlsanity.com/paper/2604.20920/) (2026)
- [Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability](https://mlsanity.com/paper/2311.04449/) (2023)
- [Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond](https://mlsanity.com/paper/2410.12982/) (2024)
- [Maglev: Sliding Recurrent Memory](https://mlsanity.com/paper/2608.02870/) (2026)
- [Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling](https://mlsanity.com/paper/2408.09453/) (2024)
- [MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling](https://mlsanity.com/paper/2512.23824/) (2025)
- [Don't Pay Attention](https://mlsanity.com/paper/2506.11305/) (2025)
- [Bridging Performance Gaps for ECG Foundation Models: A Post-Training Strategy](https://mlsanity.com/paper/2509.12991/) (2025)
- [Wavelet Predictive Representations for Non-Stationary Reinforcement Learning](https://mlsanity.com/paper/2510.04507/) (2025)
- [Detecting Structural Heart Disease from Electrocardiograms via a Generalized Additive Model of Interpretable Foundation-Model Predictors](https://mlsanity.com/paper/2603.02616/) (2026)
- [PDE-SSM: A Spectral State Space Approach to Spatial Mixing in Diffusion Transformers](https://mlsanity.com/paper/2603.13663/) (2026)
- [MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution](https://mlsanity.com/paper/2609.35664/) (2026)

## Abstract (arXiv)

Efficiently capturing the long-range patterns in sequential data sources salient to a given task -- such as classification and generative modeling -- poses a fundamental challenge. Popular approaches in the space tradeoff between the memory burden of brute-force enumeration and comparison, as in transformers, the computational burden of complicated sequential dependencies, as in recurrent neural networks, or the parameter burden of convolutional networks with many or large filters. We instead take inspiration from wavelet-based multiresolution analysis to define a new building block for sequence modeling, which we call a MultiresLayer. The key component of our model is the multiresolution convolution, capturing multiscale trends in the input sequence. Our MultiresConv can be implemented with shared filters across a dilated causal convolution tree. Thus it garners the computational advantages of convolutional networks and the principled theoretical motivation of wavelet decompositions. Our MultiresLayer is straightforward to implement, requires significantly fewer parameters, and maintains at most a O(Nlog N) memory footprint for a length N sequence. Yet, by stacking such layers, our model yields state-of-the-art performance on a number of sequence classification and autoregressive density estimation tasks using CIFAR-10, ListOps, and PTB-XL datasets.