mlsanityData to 14 Oct 2026

TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction

arXiv 2110.14392 ↗First posted 27 Oct 2021cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

While recurrent neural networks (RNNs) demonstrate outstanding capabilities for future video frame prediction, they model dynamics in a discrete time space, i.e., they predict the frames sequentially with a fixed temporal step. RNNs are therefore prone to accumulate the error as the number of future frames increases. In contrast, partial differential equations (PDEs) model physical phenomena like dynamics in a continuous time space. However, the estimated PDE for frame forecasting needs to be numerically solved, which is done by discretization of the PDE and diminishes most of the advantages compared to discrete models. In this work, we, therefore, propose to approximate the motion in a video by a continuous function using the Taylor series. To this end, we introduce TaylorSwiftNet, a novel convolutional neural network that learns to estimate the higher order terms of the Taylor series for a given input video. TaylorSwiftNet can swiftly predict future frames in parallel and it allows to change the temporal resolution of the forecast frames on-the-fly. The experimental results on various datasets demonstrate the superiority of our model.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

42 of 59 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  5. 2019Hamiltonian Neural Networkscited by 712
  6. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  7. 2016Generating Videos with Scene Dynamicscited by 376
  8. 2016Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Predictioncited by 333
  9. 2016Dynamic Filter Networkscited by 319
  10. 2017PDE-Net: Learning PDEs from Datacited by 311
  11. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  12. 2017Soft-DTW: a Differentiable Loss Function for Time-Seriescited by 271
30 more
  1. 2020Scalable Gradients for Stochastic Differential Equationscited by 271
  2. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  3. 2018Stochastic Video Generation with a Learned Priorcited by 245
  4. 2018Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equationscited by 235
  5. 2018PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Networkcited by 210
  6. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  7. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 195
  8. 2017Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognitioncited by 178
  9. 2019Symplectic Recurrent Neural Networkscited by 150
  10. 2019Hamiltonian Generative Networkscited by 137
  11. 2018PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learningcited by 122
  12. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 121
  13. 2020Stochastic Latent Residual Video Predictioncited by 108
  14. 2017Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledgecited by 96
  15. 2018Data Driven Governing Equations Approximation Using Deep Neural Networkscited by 96
  16. 2018Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamicscited by 92
  17. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  18. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  19. 2019Shape and Time Distortion Loss for Training Deep Time Series Forecasting Modelscited by 70
  20. 2017Dual Motion GAN for Future-Flow Embedded Video Predictioncited by 69
  21. 2018Flow-Grounded Spatial-Temporal Video Prediction from Still Imagescited by 55
  22. 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
  23. 2017Folded Recurrent Neural Networks for Future Video Predictioncited by 44
  24. 2017Unsupervised Learning of Long-Term Motion Dynamics for Videoscited by 37
  25. 2019ODE2VAE: Deep generative second order ODEs with Bayesian neural networkscited by 36
  26. 2015Spatio-temporal video autoencoder with differentiable memorycited by 32
  27. 2018Black-box Variational Inference for Stochastic Differential Equationscited by 25
  28. 2020PDE-Driven Spatiotemporal Disentanglementcited by 19
  29. 2018Convolutional Neural Networks combined with Runge-Kutta Methodscited by 18
  30. 2019Frequency Domain Transformer Networks for Video Predictioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Swift Sampling: Selecting Temporal Surprises via Taylor Seriescited by 2
  2. 2025Sequence-Adaptive Video Prediction in Continuous Streams using Diffusion Noise Optimizationcited by 0