# Deep Latent Force Models: ODE-based Process Convolutions for Bayesian Deep Learning

arXiv: https://arxiv.org/abs/2311.14828 · HTML: https://mlsanity.com/paper/2311.14828/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 61 references that are in the map.
- First posted on arXiv 24 Nov 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [Efficiently Sampling Functions from Gaussian Process Posteriors](https://mlsanity.com/paper/2002.09309/) (2020)
- [Avoiding pathologies in very deep networks](https://mlsanity.com/paper/1402.5836/) (2014)
- [Random Feature Expansions for Deep Gaussian Processes](https://mlsanity.com/paper/1610.04386/) (2016)
- [Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1806.05490/) (2018)
- [Deep convolutional Gaussian processes](https://mlsanity.com/paper/1810.03052/) (2018)
- [Deep Gaussian Processes with Importance-Weighted Variational Inference](https://mlsanity.com/paper/1905.05435/) (2019)
- [Implicit Posterior Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1910.11998/) (2019)
- [Markovian Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2207.05543/) (2022)
- [Physics Informed Deep Kernel Learning](https://mlsanity.com/paper/2006.04976/) (2020)
- [Non-linear process convolutions for multi-output Gaussian processes](https://mlsanity.com/paper/1810.04632/) (2018)
- [Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems](https://mlsanity.com/paper/1905.12090/) (2019)
- [Black-Box Inference for Non-Linear Latent Force Models](https://mlsanity.com/paper/1906.09199/) (2019)
- [Inter-domain Deep Gaussian Processes](https://mlsanity.com/paper/2011.00415/) (2020)
- [Compositional uncertainty in deep Gaussian processes](https://mlsanity.com/paper/1909.07698/) (2019)
- [Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Features](https://mlsanity.com/paper/2106.05960/) (2021)
- [On the inability of Gaussian process regression to optimally learn compositional functions](https://mlsanity.com/paper/2205.07764/) (2022)
- [Constraining the Dynamics of Deep Probabilistic Models](https://mlsanity.com/paper/1802.05680/) (2018)

## Built on by (most cited first, 1 of 1)

- [Adaptive RKHS Fourier Features for Compositional Gaussian Process Models](https://mlsanity.com/paper/2407.01856/) (2024)

## Abstract (arXiv)

Modelling the behaviour of highly nonlinear dynamical systems with robust uncertainty quantification is a challenging task which typically requires approaches specifically designed to address the problem at hand. We introduce a domain-agnostic model to address this issue termed the deep latent force model (DLFM), a deep Gaussian process with physics-informed kernels at each layer, derived from ordinary differential equations using the framework of process convolutions. Two distinct formulations of the DLFM are presented which utilise weight-space and variational inducing points-based Gaussian process approximations, both of which are amenable to doubly stochastic variational inference. We present empirical evidence of the capability of the DLFM to capture the dynamics present in highly nonlinear real-world multi-output time series data. Additionally, we find that the DLFM is capable of achieving comparable performance to a range of non-physics-informed probabilistic models on benchmark univariate regression tasks. We also empirically assess the negative impact of the inducing points framework on the extrapolation capabilities of LFM-based models.