# Neural signature kernels as infinite-width-depth-limits of controlled ResNets

arXiv: https://arxiv.org/abs/2303.17671 · HTML: https://mlsanity.com/paper/2303.17671/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Builds on (most cited first, 30 of 33)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [On Neural Differential Equations](https://mlsanity.com/paper/2202.02435/) (2022)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes](https://mlsanity.com/paper/1910.12478/) (2019)
- [Tensor Programs II: Neural Tangent Kernel for Any Architecture](https://mlsanity.com/paper/2006.14548/) (2020)
- [Deep Convolutional Networks as shallow Gaussian Processes](https://mlsanity.com/paper/1808.05587/) (2018)
- [Kernels for sequentially ordered data](https://mlsanity.com/paper/1601.08169/) (2016)
- [Deep Signature Transforms](https://mlsanity.com/paper/1905.08494/) (2019)
- [The Recurrent Neural Tangent Kernel](https://mlsanity.com/paper/2006.10246/) (2020)
- [The Signature Kernel is the solution of a Goursat PDE](https://mlsanity.com/paper/2006.14794/) (2020)
- [Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamics](https://mlsanity.com/paper/2110.10249/) (2021)
- [Neural Controlled Differential Equations for Online Prediction Tasks](https://mlsanity.com/paper/2106.11028/) (2021)
- [The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization](https://mlsanity.com/paper/2206.02768/) (2022)
- [Distribution Regression for Sequential Data](https://mlsanity.com/paper/2006.05805/) (2020)
- [Higher Order Kernel Mean Embeddings to Capture Filtrations of Stochastic Processes](https://mlsanity.com/paper/2109.03582/) (2021)
- [Discrete-time signatures and randomness in reservoir computing](https://mlsanity.com/paper/2010.14615/) (2020)
- [Framing RNN as a kernel method: A neural ODE approach](https://mlsanity.com/paper/2106.01202/) (2021)
- [SigGPDE: Scaling Sparse Gaussian Processes on Sequential Data](https://mlsanity.com/paper/2105.04211/) (2021)
- [On the infinite-depth limit of finite-width neural networks](https://mlsanity.com/paper/2210.00688/) (2022)
- [Policy Analysis using Synthetic Controls in Continuous-Time](https://mlsanity.com/paper/2102.01577/) (2021)
- [Scaling ResNets in the Large-depth Regime](https://mlsanity.com/paper/2206.06929/) (2022)
- [Reservoir Computing meets Recurrent Kernels and Structured Transforms](https://mlsanity.com/paper/2006.07310/) (2020)
- [SK-Tree: a systematic malware detection algorithm on streaming trees via the signature kernel](https://mlsanity.com/paper/2102.07904/) (2021)
- [Scaling Properties of Deep Residual Networks](https://mlsanity.com/paper/2105.12245/) (2021)

## Built on by (most cited first, 25 of 25)

- [Theoretical Foundations of Deep Selective State-Space Models](https://mlsanity.com/paper/2402.19047/) (2024)
- [Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit](https://mlsanity.com/paper/2309.16620/) (2023)
- [Infinite Limits of Multi-head Transformer Dynamics](https://mlsanity.com/paper/2405.15712/) (2024)
- [Non-adversarial training of Neural SDEs with signature kernel scores](https://mlsanity.com/paper/2305.16274/) (2023)
- [Signature Kernel Conditional Independence Tests in Causal Discovery for Stochastic Processes](https://mlsanity.com/paper/2402.18477/) (2024)
- [Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models](https://mlsanity.com/paper/2505.17761/) (2025)
- [Exact Gradients for Stochastic Spiking Neural Networks Driven by Rough Signals](https://mlsanity.com/paper/2405.13587/) (2024)
- [The Signature Kernel](https://mlsanity.com/paper/2305.04625/) (2023)
- [Dynamical Survival Analysis with Controlled Latent States](https://mlsanity.com/paper/2401.17077/) (2024)
- [Sparse Signature Coefficient Recovery via Kernels](https://mlsanity.com/paper/2412.08579/) (2024)
- [pySigLib -- Fast Signature-Based Computations on CPU and GPU](https://mlsanity.com/paper/2509.10613/) (2025)
- [Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks](https://mlsanity.com/paper/2310.12079/) (2023)
- [Universal randomised signatures for generative time series modelling](https://mlsanity.com/paper/2406.10214/) (2024)
- [Random Feature Representation Boosting](https://mlsanity.com/paper/2501.18283/) (2025)
- [ParallelFlow: Parallelizing Linear Transformers via Flow Discretization](https://mlsanity.com/paper/2504.00492/) (2025)
- [On the Generalization and Approximation Capacities of Neural Controlled Differential Equations](https://mlsanity.com/paper/2305.16791/) (2023)
- [Numerical Schemes for Signature Kernels](https://mlsanity.com/paper/2502.08470/) (2025)
- [Commutative Width and Depth Scaling in Deep Neural Networks](https://mlsanity.com/paper/2310.01683/) (2023)
- [Infinite-dimensional Mahalanobis Distance with Applications to Kernelized Novelty Detection](https://mlsanity.com/paper/2407.11873/) (2024)
- [Kernel Learning for Mean-Variance Trading Strategies](https://mlsanity.com/paper/2507.10701/) (2025)
- [Autonomous-Flow-Based Generation](https://mlsanity.com/paper/2511.09902/) (2025)
- [Generating Financial Time Series by Matching Random Convolutional Features](https://mlsanity.com/paper/2606.05138/) (2026)
- [Log-PDE Methods for Rough Signature Kernels](https://mlsanity.com/paper/2404.02926/) (2024)
- [Novelty detection on path space](https://mlsanity.com/paper/2512.03243/) (2025)
- [Random Controlled Differential Equations](https://mlsanity.com/paper/2512.23670/) (2025)

## Abstract (arXiv)

Motivated by the paradigm of reservoir computing, we consider randomly initialized controlled ResNets defined as Euler-discretizations of neural controlled differential equations (Neural CDEs), a unified architecture which enconpasses both RNNs and ResNets. We show that in the infinite-width-depth limit and under proper scaling, these architectures converge weakly to Gaussian processes indexed on some spaces of continuous paths and with kernels satisfying certain partial differential equations (PDEs) varying according to the choice of activation function, extending the results of Hayou (2022); Hayou & Yang (2023) to the controlled and homogeneous case. In the special, homogeneous, case where the activation is the identity, we show that the equation reduces to a linear PDE and the limiting kernel agrees with the signature kernel of Salvi et al. (2021a). We name this new family of limiting kernels neural signature kernels. Finally, we show that in the infinite-depth regime, finite-width controlled ResNets converge in distribution to Neural CDEs with random vector fields which, depending on whether the weights are shared across layers, are either time-independent and Gaussian or behave like a matrix-valued Brownian motion.