mlsanityData to 7 Oct 2026

CKConv: Continuous Kernel Convolution For Sequential Data

arXiv 2102.02611 ↗First posted 4 Feb 2021cs.LGcited by 117

Not on the map. Why?

Abstract (arXiv)

Conventional neural architectures for sequential data present important limitations. Recurrent networks suffer from exploding and vanishing gradients, small effective memory horizons, and must be trained sequentially. Convolutional networks are unable to handle sequences of unknown size and their memory horizon must be defined a priori. In this work, we show that all these problems can be solved by formulating convolutional kernels in CNNs as continuous functions. The resulting Continuous Kernel Convolution (CKConv) allows us to model arbitrarily long sequences in a parallel manner, within a single operation, and without relying on any form of recurrence. We show that Continuous Kernel Convolutional Networks (CKCNNs) obtain state-of-the-art results in multiple datasets, e.g., permuted MNIST, and, thanks to their continuous nature, are able to handle non-uniformly sampled datasets and irregularly-sampled data natively. CKCNNs match or perform better than neural ODEs designed for these purposes in a faster and simpler manner.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 49 of its 82 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

49 of 82 references.

References · 49

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  4. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  5. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  6. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  9. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  10. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  11. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  12. 2016Language Modeling with Gated Convolutional Networkscited by 688
37 more
  1. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  2. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  3. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  4. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  5. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  6. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 584
  7. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 569
  8. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 548
  9. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  10. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  11. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  12. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  13. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  14. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 366
  15. 2018The UEA multivariate time series classification archive, 2018cited by 330
  16. 2017Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphscited by 307
  17. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  18. 2020Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Datacited by 248
  19. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 230
  20. 2019Complexity of Linear Regions in Deep Networkscited by 166
  21. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  22. 2021Deep Parametric Continuous Convolutional Neural Networkscited by 161
  23. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  24. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  25. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  26. 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
  27. 2019Interpolated Convolutional Networks for 3D Point Cloud Understandingcited by 86
  28. 2017Dilated Recurrent Neural Networkscited by 83
  29. 2016Very Deep Convolutional Neural Networks for Raw Waveformscited by 68
  30. 2020DeepSphere: a graph-based spherical CNNcited by 65
  31. 2013How to Construct Deep Recurrent Neural Networkscited by 64
  32. 2020Group Equivariant Stand-Alone Self-Attention For Visioncited by 63
  33. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  34. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 56
  35. 2018Mad Max: Affine Spline Insights into Deep Learningcited by 54
  36. 2018Trellis Networks for Sequence Modelingcited by 50
  37. 2020Do RNN and LSTM have Long Memory?cited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 21; 2023: 35; 2024: 31; 2025: 15; 2026: 7.

117 papers.

Sort:

By area:

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  4. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  6. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  7. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  8. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  9. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  10. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  11. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  12. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
38 more
  1. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  2. 2022Liquid Structural State-Space Modelscited by 134
  3. 2024The Hidden Attention of Mamba Modelscited by 88
  4. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  5. 2024State Space Models for Event Camerascited by 75
  6. 2022LargeKernel3D: Scaling up Kernels in 3D Sparse CNNscited by 71
  7. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  8. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  9. 2022SKFlow: Learning Optical Flow with Super Kernelscited by 56
  10. 2023Spatial Functa: Scaling Functa to ImageNet Classification and Generationcited by 55
  11. 2024A Survey on Visual Mambacited by 50
  12. 2021Learning Partial Equivariances from Datacited by 44
  13. 2024A Survey on Transformer Compressioncited by 44
  14. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 43
  15. 2022Relaxing Equivariance Constraints with Non-stationary Continuous Filterscited by 41
  16. 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spacescited by 41
  17. 2023Fast, Expressive SE(n) Equivariant Networks through Weight-Sharing in Position-Orientation Spacecited by 40
  18. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 38
  19. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  20. 2022Deep Latent State Space Models for Time-Series Generationcited by 37
  21. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
  22. 2023Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutionscited by 24
  23. 2024Clifford-Steerable Convolutional Neural Networkscited by 24
  24. 2021Exploiting Redundancy: Separable Group Convolutional Networks on Lie Groupscited by 22
  25. 2021Dilated convolution with learnable spacingscited by 20
  26. 2023PatchMixer: A Patch-Mixing Architecture for Long-Term Time Series Forecastingcited by 20
  27. 2023Sequence Modeling with Multiresolution Convolutional Memorycited by 19
  28. 2023Modelling Long Range Dependencies in ND: From Task-Specific to a General Purpose CNNcited by 18
  29. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  30. 2024State-Free Inference of State-Space Models: The Transfer Function Approachcited by 18
  31. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  32. 2022Towards a General Purpose CNN for Long Range Dependencies in NDcited by 16
  33. 2024Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patchingcited by 16
  34. 2024Tuning Frequency Bias of State Space Modelscited by 16
  35. 2024Simba: Mamba augmented U-ShiftGCN for Skeletal Action Recognition in Videoscited by 13
  36. 2021Neural Waveshaping Synthesiscited by 12
  37. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  38. 2023Lie Group Decompositions for Equivariant Neural Networkscited by 12

50 most cited of 117 All 117 (JSON)