Taylorformer: Probabilistic Modelling for Random Processes including Time Series
Not on the map. Why?
Abstract (arXiv)
We propose the Taylorformer for random processes such as time series. Its two key components are: 1) the LocalTaylor wrapper which adapts Taylor approximations (used in dynamical systems) for use in neural network-based probabilistic models, and 2) the MHA-X attention block which makes predictions in a way inspired by how Gaussian Processes' mean predictions are linear smoothings of contextual data. Taylorformer outperforms the state-of-the-art in terms of log-likelihood on 5/6 classic Neural Process tasks such as meta-learning 1D functions, and has at least a 14% MSE improvement on forecasting tasks, including electricity, oil temperatures and exchange rates. Taylorformer approximates a consistent stochastic process and provides uncertainty-aware predictions. Our code is provided in the supplementary material.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 27 references that are in the map. First posted on arXiv 30 May 2023.
Builds on
23 of 27 references.
References · 23
- 2017Attention Is All You Needcited by 59,932
- 2018Neural Ordinary Differential Equationscited by 3,733
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2020Big Bird: Transformers for Longer Sequencescited by 1,673
- 2020Reformer: The Efficient Transformercited by 1,658
- 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
- 2020Rethinking Attention with Performerscited by 1,584
- 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 869
- 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
- 2018Conditional Neural Processescited by 421
11 more
- 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
- 2017EMNIST: an extension of MNIST to handwritten letterscited by 212
- 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
- 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
- 2019Convolutional Conditional Neural Processescited by 139
- 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
- 2019Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Modelcited by 45
- 2020Bootstrapping Neural Processescited by 40
- 2020Hierarchical Deep Learning of Multiscale Differential Equation Time-Stepperscited by 37
- 2020Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variablescited by 33
- 2022Using Probabilistic Machine Learning to Better Model Temporal Patterns in Parameterizations: a case study with the Lorenz 96 modelcited by 5
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.