mlsanityData to 14 Oct 2026

Layered Controllable Video Generation

arXiv 2111.12747 ↗First posted 24 Nov 2021cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce layered controllable video generation, where we, without any supervision, decompose the initial frame of a video into foreground and background layers, with which the user can control the video generation process by simply manipulating the foreground mask. The key challenges are the unsupervised foreground-background separation, which is ambiguous, and ability to anticipate user manipulations with access to only raw video sequences. We address these challenges by proposing a two-stage learning procedure. In the first stage, with the rich set of losses and dynamic foreground size prior, we learn how to separate the frame into foreground and background layers and, conditioned on these layers, how to generate the next frame using VQ-VAE generator. In the second stage, we fine-tune this network to anticipate edits to the mask, by fitting (parameterized) control to the mask from future frame. We demonstrate the effectiveness of this learning and the more granular control mechanism, while illustrating state-of-the-art performance on two benchmark datasets. We provide a video abstract as well as some video results on https://gabriel-huang.github.io/layered_controllable_video_generation

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 54 of its 76 references that are in the map. First posted on arXiv 24 Nov 2021.

Builds on

54 of 76 references.

References · 54

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2017Neural Discrete Representation Learningcited by 4,712
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  6. 2013Auto-Encoding Variational Bayescited by 4,053
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2015Spatial Transformer Networkscited by 1,429
  11. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
38 more
  1. 2019CNN-generated images are surprisingly easy to spot... for nowcited by 847
  2. 2016Pixel Recurrent Neural Networkscited by 816
  3. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  4. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  5. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  6. 2018Video-to-Video Synthesiscited by 390
  7. 2016Generating Videos with Scene Dynamicscited by 376
  8. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  9. 2021Video Transformer Networkcited by 272
  10. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  11. 2018Stochastic Video Generation with a Learned Priorcited by 245
  12. 2017Stochastic Variational Video Predictioncited by 234
  13. 2019Scaling Autoregressive Video Modelscited by 190
  14. 2018Stochastic Adversarial Video Predictioncited by 173
  15. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  16. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  17. 2019DwNet: Dense warp-based network for pose-guided human video generationcited by 115
  18. 2016Video Pixel Networkscited by 108
  19. 2020Stochastic Latent Residual Video Predictioncited by 108
  20. 2020Latent Video Transformercited by 106
  21. 2018Image Generation from Layoutcited by 105
  22. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  23. 2017The Pose Knows: Video Forecasting by Generating Pose Futurescited by 88
  24. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  25. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  26. 2017Recurrent Environment Simulatorscited by 68
  27. 2018Pose Guided Human Video Generationcited by 67
  28. 2021Playable Video Generationcited by 64
  29. 2019Emergence of Object Segmentation in Perturbed Generative Modelscited by 57
  30. 2019Video Generation from Single Semantic Label Mapcited by 54
  31. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  32. 2021Eyes Tell All: Irregular Pupil Shapes Reveal GAN-generated Facescited by 48
  33. 2018Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANscited by 36
  34. 2019Unsupervised Keypoint Learning for Guiding Class-Conditional Video Predictioncited by 36
  35. 2021MarioNette: Self-Supervised Sprite Learningcited by 35
  36. 2018Probabilistic Video Generation using Holistic Attribute Controlcited by 24
  37. 2018The Visual Centrifuge: Model-Free Layered Video Representationscited by 23
  38. 2015Anticipating Visual Representations from Unlabeled Videocited by 14

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 1; 2025: 2.

7 papers.

  1. 2024Genie: Generative Interactive Environmentscited by 747
  2. 2023A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decisioncited by 11
  3. 2023INVE: Interactive Neural Video Editingcited by 10
  4. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  5. 2022Controllable Video Generation through Global and Local Motion Dynamicscited by 6
  6. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  7. 2023Fast Fourier Inception Networks for Occluded Video Predictioncited by 1