# Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns

arXiv: https://arxiv.org/abs/1911.11294 · HTML: https://mlsanity.com/paper/1911.11294/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2019.

## Built on by (most cited first, 13 of 13)

- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [Latent Image Animator: Learning to Animate Images via Latent Space Navigation](https://mlsanity.com/paper/2203.09043/) (2022)
- [Vlogger: Make Your Dream A Vlog](https://mlsanity.com/paper/2401.09414/) (2024)
- [LEO: Generative Latent Image Animator for Human Video Synthesis](https://mlsanity.com/paper/2305.03989/) (2023)
- [Semi-Supervised Video Deraining with Dynamical Rain Generator](https://mlsanity.com/paper/2103.07939/) (2021)
- [Blind Image Super-resolution with Elaborate Degradation Modeling on Noise and Kernel](https://mlsanity.com/paper/2107.00986/) (2021)
- [VideoWorld 2: Learning Transferable Knowledge from Real-world Videos](https://mlsanity.com/paper/2602.10102/) (2026)
- [A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference](https://mlsanity.com/paper/2301.09300/) (2023)
- [Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE](https://mlsanity.com/paper/2303.05323/) (2023)
- [Latent Space Energy-based Neural ODEs](https://mlsanity.com/paper/2409.03845/) (2024)
- [Video Reenactment as Inductive Bias for Content-Motion Disentanglement](https://mlsanity.com/paper/2102.00324/) (2021)
- [Likelihood-Based Generative Radiance Field with Latent Space Energy-Based Model for 3D-Aware Disentangled Image Representation](https://mlsanity.com/paper/2304.07918/) (2023)
- [N-Tree Diffusion for Long-Horizon Wildfire Risk Forecasting](https://mlsanity.com/paper/2603.07361/) (2026)

## Abstract (arXiv)

Dynamic patterns are characterized by complex spatial and motion patterns. Understanding dynamic patterns requires a disentangled representational model that separates the factorial components. A commonly used model for dynamic patterns is the state space model, where the state evolves over time according to a transition model and the state generates the observed image frames according to an emission model. To model the motions explicitly, it is natural for the model to be based on the motions or the displacement fields of the pixels. Thus in the emission model, we let the hidden state generate the displacement field, which warps the trackable component in the previous image frame to generate the next frame while adding a simultaneously emitted residual image to account for the change that cannot be explained by the deformation. The warping of the previous image is about the trackable part of the change of image frame, while the residual image is about the intrackable part of the image. We use a maximum likelihood algorithm to learn the model that iterates between inferring latent noise vectors that drive the transition model and updating the parameters given the inferred latent vectors. Meanwhile we adopt a regularization term to penalize the norms of the residual images to encourage the model to explain the change of image frames by trackable motion. Unlike existing methods on dynamic patterns, we learn our model in unsupervised setting without ground truth displacement fields. In addition, our model defines a notion of intrackability by the separation of warped component and residual component in each image frame. We show that our method can synthesize realistic dynamic pattern, and disentangling appearance, trackable and intrackable motions. The learned models are useful for motion transfer, and it is natural to adopt it to define and measure intrackability of a dynamic pattern.