# Framing RNN as a kernel method: A neural ODE approach

arXiv: https://arxiv.org/abs/2106.01202 · HTML: https://mlsanity.com/paper/2106.01202/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 61 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series](https://mlsanity.com/paper/1905.12374/) (2019)
- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [Neural Rough Differential Equations for Long Time Series](https://mlsanity.com/paper/2009.08295/) (2020)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [A Primer on the Signature Method in Machine Learning](https://mlsanity.com/paper/1603.03788/) (2016)
- [Lipschitz Recurrent Neural Networks](https://mlsanity.com/paper/2006.12070/) (2020)
- [Kernels for sequentially ordered data](https://mlsanity.com/paper/1601.08169/) (2016)
- [Learning Differential Equations that are Easy to Solve](https://mlsanity.com/paper/2007.04504/) (2020)
- [Deep Signature Transforms](https://mlsanity.com/paper/1905.08494/) (2019)
- [Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU](https://mlsanity.com/paper/2001.00706/) (2020)
- [On Generalization Bounds of a Family of Recurrent Neural Networks](https://mlsanity.com/paper/1910.12947/) (2019)
- [The iisignature library: efficient calculation of iterated-integral signatures and log signatures](https://mlsanity.com/paper/1802.08252/) (2018)
- [POPQORN: Quantifying Robustness of Recurrent Neural Networks](https://mlsanity.com/paper/1905.07387/) (2019)
- [Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization](https://mlsanity.com/paper/1803.09327/) (2018)
- [Embedding and learning with signatures](https://mlsanity.com/paper/1911.13211/) (2019)
- [Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition](https://mlsanity.com/paper/1707.03993/) (2017)
- [DeepWriterID: An End-to-end Online Text-independent Writer Identification System](https://mlsanity.com/paper/1508.04945/) (2015)
- [Learning stochastic differential equations using RNN with log signature features](https://mlsanity.com/paper/1908.08286/) (2019)
- [Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory](https://mlsanity.com/paper/2006.11052/) (2020)
- [Sample Complexity Bounds for Recurrent Neural Networks with Application to Combinatorial Graph Problems](https://mlsanity.com/paper/1901.10289/) (2019)

## Built on by (most cited first, 23 of 23)

- [Theoretical Foundations of Deep Selective State-Space Models](https://mlsanity.com/paper/2402.19047/) (2024)
- [Neural signature kernels as infinite-width-depth-limits of controlled ResNets](https://mlsanity.com/paper/2303.17671/) (2023)
- [Generalization bounds for neural ordinary differential equations and deep residual networks](https://mlsanity.com/paper/2305.06648/) (2023)
- [Scaling ResNets in the Large-depth Regime](https://mlsanity.com/paper/2206.06929/) (2022)
- [Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules](https://mlsanity.com/paper/2206.01649/) (2022)
- [New directions in the applications of rough path theory](https://mlsanity.com/paper/2302.04586/) (2023)
- [Random Fourier Signature Features](https://mlsanity.com/paper/2311.12214/) (2023)
- [The Signature Kernel](https://mlsanity.com/paper/2305.04625/) (2023)
- [Dynamical Survival Analysis with Controlled Latent States](https://mlsanity.com/paper/2401.17077/) (2024)
- [Rademacher Complexity of Neural ODEs via Chen-Fliess Series](https://mlsanity.com/paper/2401.16655/) (2024)
- [Learning the Dynamics of Sparsely Observed Interacting Systems](https://mlsanity.com/paper/2301.11647/) (2023)
- [On the Generalization and Approximation Capacities of Neural Controlled Differential Equations](https://mlsanity.com/paper/2305.16791/) (2023)
- [Numerical Schemes for Signature Kernels](https://mlsanity.com/paper/2502.08470/) (2025)
- [Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraints](https://mlsanity.com/paper/2405.20278/) (2024)
- [Realization Theory Of Recurrent Neural ODEs Using Polynomial System Embeddings](https://mlsanity.com/paper/2205.11989/) (2022)
- [On Tuning Neural ODE for Stability, Consistency and Faster Convergence](https://mlsanity.com/paper/2312.01657/) (2023)
- [Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba](https://mlsanity.com/paper/2406.16722/) (2024)
- [Sliding-Window Signatures for Time Series: Application to Electricity Demand Forecasting](https://mlsanity.com/paper/2510.12337/) (2025)
- [Noisy Learning for Neural ODEs Acts as a Robustness Locus Widening](https://mlsanity.com/paper/2206.08237/) (2022)
- [PAC bounds of continuous Linear Parameter-Varying systems related to neural ODEs](https://mlsanity.com/paper/2307.03630/) (2023)
- [Log-PDE Methods for Rough Signature Kernels](https://mlsanity.com/paper/2404.02926/) (2024)
- [A finite-sample generalization bound for stable LPV systems](https://mlsanity.com/paper/2405.10054/) (2024)
- [Upper Generalization Bounds for Neural Oscillators](https://mlsanity.com/paper/2603.09742/) (2026)

## Abstract (arXiv)

Building on the interpretation of a recurrent neural network (RNN) as a continuous-time neural differential equation, we show, under appropriate conditions, that the solution of a RNN can be viewed as a linear function of a specific feature set of the input sequence, known as the signature. This connection allows us to frame a RNN as a kernel method in a suitable reproducing kernel Hilbert space. As a consequence, we obtain theoretical guarantees on generalization and stability for a large class of recurrent networks. Our results are illustrated on simulated datasets.