mlsanityData to 7 Oct 2026

Framing RNN as a kernel method: A neural ODE approach

arXiv 2106.01202 ↗First posted 2 Jun 2021stat.ML cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Building on the interpretation of a recurrent neural network (RNN) as a continuous-time neural differential equation, we show, under appropriate conditions, that the solution of a RNN can be viewed as a linear function of a specific feature set of the input sequence, known as the signature. This connection allows us to frame a RNN as a kernel method in a suitable reproducing kernel Hilbert space. As a consequence, we obtain theoretical guarantees on generalization and stability for a large class of recurrent networks. Our results are illustrated on simulated datasets.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 28 of its 61 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

28 of 61 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2018Neural Ordinary Differential Equationscited by 3,733
  6. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  7. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  8. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  9. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 230
  10. 2018To understand deep learning we need to understand kernel learningcited by 186
  11. 2020Neural Rough Differential Equations for Long Time Seriescited by 140
  12. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
16 more
  1. 2016A Primer on the Signature Method in Machine Learningcited by 100
  2. 2020Lipschitz Recurrent Neural Networkscited by 88
  3. 2016Kernels for sequentially ordered datacited by 83
  4. 2020Learning Differential Equations that are Easy to Solvecited by 81
  5. 2019Deep Signature Transformscited by 75
  6. 2020Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPUcited by 45
  7. 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
  8. 2018The iisignature library: efficient calculation of iterated-integral signatures and log signaturescited by 38
  9. 2019POPQORN: Quantifying Robustness of Recurrent Neural Networkscited by 36
  10. 2018Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterizationcited by 35
  11. 2019Embedding and learning with signaturescited by 29
  12. 2017Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognitioncited by 20
  13. 2015DeepWriterID: An End-to-end Online Text-independent Writer Identification Systemcited by 14
  14. 2019Learning stochastic differential equations using RNN with log signature featurescited by 13
  15. 2020Understanding Recurrent Neural Networks Using Nonequilibrium Response Theorycited by 10
  16. 2019Sample Complexity Bounds for Recurrent Neural Networks with Application to Combinatorial Graph Problemscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 9; 2024: 7; 2025: 2; 2026: 1.

23 papers.

Sort:

By area:

  1. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
  2. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  3. 2023Generalization bounds for neural ordinary differential equations and deep residual networkscited by 21
  4. 2022Scaling ResNets in the Large-depth Regimecited by 18
  5. 2022Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rulescited by 15
  6. 2023New directions in the applications of rough path theorycited by 12
  7. 2023Random Fourier Signature Featurescited by 9
  8. 2023The Signature Kernelcited by 8
  9. 2024Dynamical Survival Analysis with Controlled Latent Statescited by 7
  10. 2024Rademacher Complexity of Neural ODEs via Chen-Fliess Seriescited by 5
  11. 2023Learning the Dynamics of Sparsely Observed Interacting Systemscited by 3
  12. 2023On the Generalization and Approximation Capacities of Neural Controlled Differential Equationscited by 3
11 more
  1. 2025Numerical Schemes for Signature Kernelscited by 3
  2. 2024Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraintscited by 2
  3. 2022Realization Theory Of Recurrent Neural ODEs Using Polynomial System Embeddingscited by 1
  4. 2023On Tuning Neural ODE for Stability, Consistency and Faster Convergencecited by 1
  5. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mambacited by 1
  6. 2025Sliding-Window Signatures for Time Series: Application to Electricity Demand Forecastingcited by 1
  7. 2022Noisy Learning for Neural ODEs Acts as a Robustness Locus Wideningcited by 0
  8. 2023PAC bounds of continuous Linear Parameter-Varying systems related to neural ODEscited by 0
  9. 2024Log-PDE Methods for Rough Signature Kernelscited by 0
  10. 2024A finite-sample generalization bound for stable LPV systemscited by 0
  11. 2026Upper Generalization Bounds for Neural Oscillatorscited by 0