# Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patterns

arXiv: https://arxiv.org/abs/1909.11975 · HTML: https://mlsanity.com/paper/1909.11975/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2019.

## Built on by (most cited first, 22 of 22)

- [Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement](https://mlsanity.com/paper/2301.03028/) (2023)
- [Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction](https://mlsanity.com/paper/2112.13528/) (2021)
- [HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences](https://mlsanity.com/paper/2210.01753/) (2022)
- [A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model](https://mlsanity.com/paper/2205.06924/) (2022)
- [Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution](https://mlsanity.com/paper/2103.14230/) (2021)
- [ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation](https://mlsanity.com/paper/2111.15483/) (2021)
- [Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood](https://mlsanity.com/paper/2309.05153/) (2023)
- [Energy-Based Residual Latent Transport for Unsupervised Point Cloud Completion](https://mlsanity.com/paper/2211.06820/) (2022)
- [Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation](https://mlsanity.com/paper/2103.04285/) (2021)
- [Energy-Based Generative Cooperative Saliency Prediction](https://mlsanity.com/paper/2106.13389/) (2021)
- [Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE](https://mlsanity.com/paper/2303.05323/) (2023)
- [Combating Mode Collapse in GANs via Manifold Entropy Estimation](https://mlsanity.com/paper/2208.12055/) (2022)
- [EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models](https://mlsanity.com/paper/2105.04771/) (2021)
- [An Energy-Based Prior for Generative Saliency](https://mlsanity.com/paper/2204.08803/) (2022)
- [Maximizing Discrimination Capability of Knowledge Distillation with Energy Function](https://mlsanity.com/paper/2311.14334/) (2023)
- [Energy Aligning for Biased Models](https://mlsanity.com/paper/2106.03343/) (2021)
- [Likelihood-Based Generative Radiance Field with Latent Space Energy-Based Model for 3D-Aware Disentangled Image Representation](https://mlsanity.com/paper/2304.07918/) (2023)
- [Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning](https://mlsanity.com/paper/2605.16776/) (2026)
- [An Energy-Based View of Graph Neural Networks](https://mlsanity.com/paper/2104.13492/) (2021)
- [STANLEY: Stochastic Gradient Anisotropic Langevin Dynamics for Learning Energy-Based Models](https://mlsanity.com/paper/2310.12667/) (2023)
- [A Hybrid Generative and Discriminative PointNet on Unordered Point Sets](https://mlsanity.com/paper/2404.12925/) (2024)
- [Learning a Maximum Entropy Model for Visual Textures using Diffusion](https://mlsanity.com/paper/2606.17342/) (2026)

## Abstract (arXiv)

Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action patterns that are non-stationary in either spatial or temporal domain. We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns. The model defines a probability distribution on the video sequence, and the log probability is defined by a spatial-temporal ConvNet that consists of multiple layers of spatial-temporal filters to capture spatial-temporal patterns of different scales. The model can be learned from the training video sequences by an "analysis by synthesis" learning algorithm that iterates the following two steps. Step 1 synthesizes video sequences from the currently learned model. Step 2 then updates the model parameters based on the difference between the synthesized video sequences and the observed training sequences. We show that the learning algorithm can synthesize realistic dynamic patterns. We also show that it is possible to learn the model from incomplete training sequences with either occluded pixels or missing frames, so that model learning and pattern completion can be accomplished simultaneously.