mlsanityData to 7 Oct 2026

CCVS: Context-aware Controllable Video Synthesis

arXiv 2107.08037 ↗First posted 16 Jul 2021cs.CVcited by 68

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This presentation introduces a self-supervised learning approach to the synthesis of new video clips from old ones, with several new key elements for improved spatial resolution and realism: It conditions the synthesis process on contextual information for temporal continuity and ancillary information for fine control. The prediction model is doubly autoregressive, in the latent space of an autoencoder for forecasting, and in image space for updating contextual information, which is also used to enforce spatio-temporal consistency through a learnable optical flow module. Adversarial training of the autoencoder in the appearance and temporal domains is used to further improve the realism of its output. A quantizer inserted between the encoder and the transformer in charge of forecasting future frames in latent space (and its inverse inserted between the transformer and the decoder) adds even more flexibility by affording simple mechanisms for handling multimodal ancillary information for controlling the synthesis process (eg, a few sample frames, an audio track, a trajectory in image space) and taking into account the intrinsically uncertain nature of the future by allowing multiple predictions. Experiments with an implementation of the proposed approach give very good qualitative and quantitative results on multiple tasks and standard benchmarks.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 66 of its 84 references that are in the map. First posted on arXiv 16 Jul 2021.

Builds on

66 of 84 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
38 more
  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2015Spatial Transformer Networkscited by 1,423
  10. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  13. 2019Green AIcited by 620
  14. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  15. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  16. 2018Video-to-Video Synthesiscited by 388
  17. 2016Generating Videos with Scene Dynamicscited by 376
  18. 2018A Short Note about Kinetics-600cited by 331
  19. 2018Everybody Dance Nowcited by 313
  20. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  21. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  22. 2018Stochastic Video Generation with a Learned Priorcited by 244
  23. 2017Stochastic Variational Video Predictioncited by 234
  24. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  25. 2019Scaling Autoregressive Video Modelscited by 189
  26. 2020Hierarchical Multi-Scale Attention for Semantic Segmentationcited by 183
  27. 2018Stochastic Adversarial Video Predictioncited by 173
  28. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  29. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  30. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  31. 2020What Matters in Unsupervised Optical Flowcited by 124
  32. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  33. 2020Stochastic Latent Residual Video Predictioncited by 106
  34. 2020Latent Video Transformercited by 106
  35. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  36. 2019Deep Learning for Deepfakes Creation and Detection: A Surveycited by 102
  37. 2017You said that?cited by 99
  38. 2019Diversity-Sensitive Conditional Generative Adversarial Networkscited by 77

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 15; 2023: 20; 2024: 18; 2025: 11; 2026: 3.

68 papers.

Sort:

By area:

  1. 2022Video Diffusion Modelscited by 2,280
  2. 2024Genie: Generative Interactive Environmentscited by 706
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  4. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  5. 2022MAGVIT: Masked Generative Video Transformercited by 341
  6. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  7. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  8. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  9. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  10. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  11. 2023A Survey on Video Diffusion Modelscited by 203
  12. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
38 more
  1. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
  2. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  3. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  4. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  5. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  6. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  7. 2024Rolling Diffusion Modelscited by 102
  8. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 54
  9. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generationcited by 49
  10. 2022Temporally Consistent Transformers for Video Generationcited by 43
  11. 2024Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditionscited by 42
  12. 2023The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusioncited by 41
  13. 2024ProcessPainter: Learn Painting Process from Sequence Datacited by 37
  14. 2022Sound-Guided Semantic Video Generationcited by 35
  15. 2024Motion Inversion for Video Customizationcited by 30
  16. 2024Pix2Code: Learning to Compose Neural Visual Concepts as Programscited by 18
  17. 2022Context-Aware Video Reconstruction for Rolling Shutter Camerascited by 15
  18. 2023Video Diffusion Models with Local-Global Context Guidancecited by 15
  19. 2023The Age of Synthetic Realities: Challenges and Opportunitiescited by 15
  20. 2025Audio-Sync Video Generation with Multi-Stream Temporal Controlcited by 14
  21. 2022Neighbor Correspondence Matching for Flow-based Video Frame Synthesiscited by 12
  22. 2023MOSO: Decomposing MOtion, Scene and Object for Video Predictioncited by 12
  23. 2023Visual Affordance Prediction for Guiding Robot Explorationcited by 12
  24. 2023MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformerscited by 11
  25. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  26. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 11
  27. 2023Transformation vs Tradition: Artificial General Intelligence (AGI) for Arts and Humanitiescited by 9
  28. 2023Decouple Content and Motion for Conditional Image-to-Video Generationcited by 9
  29. 2022SSGVS: Semantic Scene Graph-to-Video Synthesiscited by 8
  30. 2024ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Contextcited by 8
  31. 2022INR-V: A Continuous Representation Space for Video-based Generative Taskscited by 7
  32. 2023Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformerscited by 7
  33. 2023Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generationcited by 7
  34. 2024SMITE: Segment Me In TimEcited by 7
  35. 2025Articulate That Object Part (ATOP): 3D Part Articulation via Text and Motion Personalizationcited by 7
  36. 2022WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Predictioncited by 5
  37. 2024Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generationcited by 5
  38. 2025JointTuner: Appearance-Motion Adaptive Joint Training for Customized Video Generationcited by 5

50 most cited of 68 All 68 (JSON)