mlsanityData to 7 Oct 2026

Neural signature kernels as infinite-width-depth-limits of controlled ResNets

arXiv 2303.17671 ↗First posted 30 Mar 2023math.DS cs.LG math.PRcited by 25

Not on the map. Why?

Abstract (arXiv)

Motivated by the paradigm of reservoir computing, we consider randomly initialized controlled ResNets defined as Euler-discretizations of neural controlled differential equations (Neural CDEs), a unified architecture which enconpasses both RNNs and ResNets. We show that in the infinite-width-depth limit and under proper scaling, these architectures converge weakly to Gaussian processes indexed on some spaces of continuous paths and with kernels satisfying certain partial differential equations (PDEs) varying according to the choice of activation function, extending the results of Hayou (2022); Hayou & Yang (2023) to the controlled and homogeneous case. In the special, homogeneous, case where the activation is the identity, we show that the equation reduces to a linear PDE and the limiting kernel agrees with the signature kernel of Salvi et al. (2021a). We name this new family of limiting kernels neural signature kernels. Finally, we show that in the infinite-depth regime, finite-width controlled ResNets converge in distribution to Neural CDEs with random vector fields which, depending on whether the weights are shared across layers, are either time-independent and Gaussian or behave like a matrix-valued Brownian motion.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

33 of 51 references.

References · 33

  1. 2018Neural Ordinary Differential Equationscited by 3,733
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2017Deep Neural Networks as Gaussian Processescited by 617
  4. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  5. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  6. 2022On Neural Differential Equationscited by 264
  7. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  8. 2016Deep Information Propagationcited by 194
  9. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  10. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  11. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  12. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
21 more
  1. 2016Kernels for sequentially ordered datacited by 83
  2. 2019Deep Signature Transformscited by 75
  3. 2020The Recurrent Neural Tangent Kernelcited by 67
  4. 2020The Signature Kernel is the solution of a Goursat PDEcited by 58
  5. 2021Neural Controlled Differential Equations for Online Prediction Taskscited by 46
  6. 2021Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamicscited by 46
  7. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  8. 2020Distribution Regression for Sequential Datacited by 31
  9. 2021Higher Order Kernel Mean Embeddings to Capture Filtrations of Stochastic Processescited by 27
  10. 2020Discrete-time signatures and randomness in reservoir computingcited by 23
  11. 2021Framing RNN as a kernel method: A neural ODE approachcited by 23
  12. 2021SigGPDE: Scaling Sparse Gaussian Processes on Sequential Datacited by 22
  13. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  14. 2021Policy Analysis using Synthetic Controls in Continuous-Timecited by 18
  15. 2022Scaling ResNets in the Large-depth Regimecited by 18
  16. 2021SK-Tree: a systematic malware detection algorithm on streaming trees via the signature kernelcited by 16
  17. 2021Scaling Properties of Deep Residual Networkscited by 16
  18. 2020Reservoir Computing meets Recurrent Kernels and Structured Transformscited by 15
  19. 2022Applications of Signature Methods to Market Anomaly Detectioncited by 9
  20. 2022Multiscale Sensor Fusion and Continuous Control with Neural CDEscited by 4
  21. 2022Asymptotic Analysis of Deep Residual Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 9; 2025: 9; 2026: 1.

25 papers.

Sort:

By area:

  1. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
  2. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  3. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  4. 2023Non-adversarial training of Neural SDEs with signature kernel scorescited by 36
  5. 2024Signature Kernel Conditional Independence Tests in Causal Discovery for Stochastic Processescited by 18
  6. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 16
  7. 2024Exact Gradients for Stochastic Spiking Neural Networks Driven by Rough Signalscited by 9
  8. 2023The Signature Kernelcited by 8
  9. 2024Dynamical Survival Analysis with Controlled Latent Statescited by 7
  10. 2024Sparse Signature Coefficient Recovery via Kernelscited by 5
  11. 2025pySigLib -- Fast Signature-Based Computations on CPU and GPUcited by 5
  12. 2023Differential Equation Scaling Limits of Shaped and Unshaped Neural Networkscited by 4
13 more
  1. 2024Universal randomised signatures for generative time series modellingcited by 4
  2. 2025Random Feature Representation Boostingcited by 4
  3. 2023On the Generalization and Approximation Capacities of Neural Controlled Differential Equationscited by 3
  4. 2025Numerical Schemes for Signature Kernelscited by 3
  5. 2025ParallelFlow: Parallelizing Linear Transformers via Flow Discretizationcited by 3
  6. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  7. 2024Infinite-dimensional Mahalanobis Distance with Applications to Kernelized Novelty Detectioncited by 1
  8. 2025Kernel Learning for Mean-Variance Trading Strategiescited by 1
  9. 2025Autonomous-Flow-Based Generationcited by 1
  10. 2026Generating Financial Time Series by Matching Random Convolutional Featurescited by 1
  11. 2024Log-PDE Methods for Rough Signature Kernelscited by 0
  12. 2025Novelty detection on path spacecited by 0
  13. 2025Random Controlled Differential Equationscited by 0