mlsanityData to 14 Oct 2026

Disentangling Content and Motion for Text-Based Neural Video Manipulation

arXiv 2211.02980 ↗First posted 5 Nov 2022cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Giving machines the ability to imagine possible new objects or scenes from linguistic descriptions and produce their realistic renderings is arguably one of the most challenging problems in computer vision. Recent advances in deep generative models have led to new approaches that give promising results towards this goal. In this paper, we introduce a new method called DiCoMoGAN for manipulating videos with natural language, aiming to perform local and semantic edits on a video clip to alter the appearances of an object of interest. Our GAN architecture allows for better utilization of multiple observations by disentangling content and motion to enable controllable semantic edits. To this end, we introduce two tightly coupled networks: (i) a representation network for constructing a concise understanding of motion dynamics and temporally invariant content, and (ii) a translation network that exploits the extracted latent content representation to actuate the manipulation according to the target description. Our qualitative and quantitative evaluations demonstrate that DiCoMoGAN significantly outperforms existing frame-based methods, producing temporally coherent and semantically more meaningful results.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 64 of its 93 references that are in the map. First posted on arXiv 5 Nov 2022.

Builds on

64 of 93 references.

References · 64

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  8. 2018Neural Ordinary Differential Equationscited by 3,779
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  10. 2016Improved Techniques for Training GANscited by 2,970
  11. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
38 more
  1. 2014Conditional Generative Adversarial Netscited by 2,162
  2. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  3. 2017Deep Setscited by 1,421
  4. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  5. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  6. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  8. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  9. 2020Object-Centric Learning with Slot Attentioncited by 876
  10. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  11. 2016Generative Adversarial Text to Image Synthesiscited by 735
  12. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  13. 2018Disentangling by Factorisingcited by 616
  14. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  15. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 565
  16. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  17. 2019Augmented Neural ODEscited by 397
  18. 2016Generating Videos with Scene Dynamicscited by 376
  19. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  20. 2016Adversarial Feature Learningcited by 350
  21. 2018Understanding disentangling in β-VAEcited by 335
  22. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  23. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  24. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  25. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  26. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  27. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  28. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  29. 2018Stochastic Video Generation with a Learned Priorcited by 245
  30. 2016Adversarially Learned Inferencecited by 216
  31. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  32. 2019Scaling Autoregressive Video Modelscited by 190
  33. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  34. 2017Video Generation From Textcited by 154
  35. 2017Unsupervised Learning of Disentangled Representations from Videocited by 139
  36. 2020On Disentangled Representations Learned From Correlated Datacited by 94
  37. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  38. 2017Semantic Image Synthesis via Adversarial Learningcited by 87

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEscited by 3