# Unsupervised Reservoir Computing for Solving Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2108.11417 · HTML: https://mlsanity.com/paper/2108.11417/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 25 Aug 2021.

## Builds on (most cited first, 19 of 19)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Federated Bayesian Optimization via Thompson Sampling](https://mlsanity.com/paper/2010.10154/) (2020)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework](https://mlsanity.com/paper/2012.07938/) (2020)
- [A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations](https://mlsanity.com/paper/1809.02362/) (2018)
- [Hypersolvers: Toward Fast Continuous-Depth Models](https://mlsanity.com/paper/2007.09601/) (2020)
- [Solving Differential Equations Using Neural Network Solution Bundles](https://mlsanity.com/paper/2006.14372/) (2020)
- [Theory of neuromorphic computing by waves: machine learning by rogue waves, dispersive shocks, and solitons](https://mlsanity.com/paper/1912.07044/) (2019)
- [Reservoir Computing meets Recurrent Kernels and Structured Transforms](https://mlsanity.com/paper/2006.07310/) (2020)
- [Unsupervised Neural Networks for Quantum Eigenvalue Problems](https://mlsanity.com/paper/2010.05075/) (2020)
- [Efficient Optimization of Echo State Networks for Time Series Datasets](https://mlsanity.com/paper/1903.05071/) (2019)
- [Semi-supervised Neural Networks solve an inverse problem for modeling Covid-19 spread](https://mlsanity.com/paper/2010.05074/) (2020)
- [Bayesian Optimization with Unknown Search Space](https://mlsanity.com/paper/1910.13092/) (2019)
- [Unfolding recurrence by Green's functions for optimized reservoir computing](https://mlsanity.com/paper/2010.06247/) (2020)

## Built on by (most cited first, 11 of 11)

- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [One-Shot Transfer Learning of Physics-Informed Neural Networks](https://mlsanity.com/paper/2110.11286/) (2021)
- [PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Laws](https://mlsanity.com/paper/2401.07364/) (2024)
- [Physics-Informed Neural Networks for Quantum Eigenvalue Problems](https://mlsanity.com/paper/2203.00451/) (2022)
- [Transfer Learning with Physics-Informed Neural Networks for Efficient Simulation of Branched Flows](https://mlsanity.com/paper/2211.00214/) (2022)
- [DEQGAN: Learning the Loss Function for PINNs with Generative Adversarial Networks](https://mlsanity.com/paper/2209.07081/) (2022)
- [Uncertainty Quantification in Neural Differential Equations](https://mlsanity.com/paper/2111.04207/) (2021)
- [RcTorch: a PyTorch Reservoir Computing Package with Automated Hyper-Parameter Optimization](https://mlsanity.com/paper/2207.05870/) (2022)
- [A Meta-learning Approach to Reservoir Computing: Time Series Prediction with Limited Data](https://mlsanity.com/paper/2110.03722/) (2021)
- [Unsupervised Random Quantum Networks for PDEs](https://mlsanity.com/paper/2312.14975/) (2023)
- [Fourier analysis of the physics of transfer learning for data-driven subgrid-scale models of ocean turbulence](https://mlsanity.com/paper/2504.15487/) (2025)

## Abstract (arXiv)

There is a wave of interest in using unsupervised neural networks for solving differential equations. The existing methods are based on feed-forward networks, {while} recurrent neural network differential equation solvers have not yet been reported. We introduce an unsupervised reservoir computing (RC), an echo-state recurrent neural network capable of discovering approximate solutions that satisfy ordinary differential equations (ODEs). We suggest an approach to calculate time derivatives of recurrent neural network outputs without using backpropagation. The internal weights of an RC are fixed, while only a linear output layer is trained, yielding efficient training. However, RC performance strongly depends on finding the optimal hyper-parameters, which is a computationally expensive process. We use Bayesian optimization to efficiently discover optimal sets in a high-dimensional hyper-parameter space and numerically show that one set is robust and can be used to solve an ODE for different initial conditions and time ranges. A closed-form formula for the optimal output weights is derived to solve first order linear equations in a backpropagation-free learning process. We extend the RC approach by solving nonlinear system of ODEs using a hybrid optimization method consisting of gradient descent and Bayesian optimization. Evaluation of linear and nonlinear systems of equations demonstrates the efficiency of the RC ODE solver.