mlsanityData to 7 Oct 2026

Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling

arXiv 2408.09453 ↗First posted 18 Aug 2024cs.LG cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions (textttMRConv), a novel approach to parameterizing global convolutional kernels for long-sequence modelling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, textttMRConv learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D textttMRConv layers.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

42 of 50 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  9. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  10. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  11. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  12. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
30 more
  1. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  2. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  3. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  4. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  5. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  6. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
  7. 2019ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blockscited by 178
  8. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  9. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  10. 2022Liquid Structural State-Space Modelscited by 134
  11. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  12. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  13. 2020Lipschitz Recurrent Neural Networkscited by 88
  14. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  15. 2020ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgettingcited by 85
  16. 2014A Clockwork RNNcited by 70
  17. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  18. 2023UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognitioncited by 64
  19. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  20. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  21. 2018Trellis Networks for Sequence Modelingcited by 50
  22. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  23. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priorscited by 33
  24. 2021RepMLPNet: Hierarchical Vision MLP with Re-parameterized Localitycited by 32
  25. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
  26. 2023Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutionscited by 24
  27. 2023Sequence Modeling with Multiresolution Convolutional Memorycited by 19
  28. 2023Modelling Long Range Dependencies in ND: From Task-Specific to a General Purpose CNNcited by 18
  29. 2023Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layercited by 13
  30. 2024Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequencescited by 5

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Log-Linear Attentioncited by 39
  2. 2025mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modelingcited by 0