mlsanityData to 7 Oct 2026

Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE

arXiv 2303.05323 ↗First posted 9 Mar 2023cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Videos depict the change of complex dynamical systems over time in the form of discrete image sequences. Generating controllable videos by learning the dynamical system is an important yet underexplored topic in the computer vision community. This paper presents a novel framework, TiV-ODE, to generate highly controllable videos from a static image and a text caption. Specifically, our framework leverages the ability of Neural Ordinary Differential Equations~(Neural ODEs) to represent complex dynamical systems as a set of nonlinear ordinary differential equations. The resulting framework is capable of generating videos with both desired dynamics and content. Experiments demonstrate the ability of the proposed method in generating highly controllable and visually consistent videos, and its capability of modeling dynamical systems. Overall, this work is a significant step towards developing advanced controllable video generation models that can handle complex and dynamic scenes.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

41 of 54 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2018Neural Ordinary Differential Equationscited by 3,733
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  10. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  11. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  12. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
29 more
  1. 2019Augmented Neural ODEscited by 396
  2. 2016Generating Videos with Scene Dynamicscited by 376
  3. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  4. 2021Conditional Object-Centric Learning from Videocited by 247
  5. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 230
  6. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 221
  7. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  8. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  9. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  10. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  11. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  12. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
  13. 2021GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Drivingcited by 75
  14. 2020Improving Generative Imagination in Object-Centric World Modelscited by 70
  15. 2019Approximation Capabilities of Neural ODEs and Invertible Residual Networkscited by 56
  16. 2018Theory and Experiments on Vector Quantized Autoencoderscited by 55
  17. 2019Video Generation from Single Semantic Label Mapcited by 54
  18. 2016Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architecturescited by 44
  19. 2020Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equationcited by 39
  20. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  21. 2019ODE2VAE: Deep generative second order ODEs with Bayesian neural networkscited by 36
  22. 2020Universal Approximation Property of Neural Ordinary Differential Equationscited by 32
  23. 2020Shared Cross-Modal Trajectory Prediction for Autonomous Drivingcited by 31
  24. 2019Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patternscited by 22
  25. 2020Deep Slow Motion Video Reconstruction with Hybrid Imaging Systemcited by 14
  26. 2019Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patternscited by 13
  27. 2021MLIM: Vision-and-Language Model Pre-training with Masked Language and Image Modelingcited by 11
  28. 2021Object-Centric Representation Learning with Generative Spatial-Temporal Factorizationcited by 10
  29. 2021Simple Video Generation using Neural ODEscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 0; 2026: 1.

7 papers.

  1. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  2. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  3. 2023Benchmarks for Physical Reasoning AIcited by 8
  4. 2024Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunitiescited by 7
  5. 2024TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generationcited by 7
  6. 2024Guiding Video Prediction with Explicit Procedural Knowledgecited by 1
  7. 2026Reinforcement Learning: From Algorithms To Foundation Modelscited by 0