mlsanityData to 14 Oct 2026

Video-to-Video Synthesis

arXiv 1808.06601 ↗First posted 20 Aug 2018cs.CV cs.GR cs.LGcited by 390

Not on the map. Why?

Cited by 46 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e.g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video. While its image counterpart, the image-to-image synthesis problem, is a popular topic, the video-to-video synthesis problem is less explored in the literature. Without understanding temporal dynamics, directly applying existing image synthesis approaches to an input video often results in temporally incoherent videos of low visual quality. In this paper, we propose a novel video-to-video synthesis approach under the generative adversarial learning framework. Through carefully-designed generator and discriminator architectures, coupled with a spatio-temporal adversarial objective, we achieve high-resolution, photorealistic, temporally coherent video results on a diverse set of input formats including segmentation masks, sketches, and poses. Experiments on multiple benchmarks show the advantage of our method compared to strong baselines. In particular, our model is capable of synthesizing 2K resolution videos of street scenes up to 30 seconds long, which significantly advances the state-of-the-art of video synthesis. Finally, we apply our approach to future video prediction, outperforming several state-of-the-art competing systems.

Key facts. Cited by 390 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 113; 2022: 91; 2023: 72; 2024: 60; 2025: 34; 2026: 20.

390 papers.

Sort:

By area:

  1. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  2. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  3. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  4. 2022HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Videocited by 483
  5. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  6. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  7. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  8. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  9. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 287
  10. 2021Motion Representations for Articulated Animationcited by 255
  11. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  12. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
38 more
  1. 2023A Survey on Video Diffusion Modelscited by 206
  2. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 203
  3. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 197
  4. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 181
  5. 2021Audio-Driven Emotional Video Portraitscited by 180
  6. 2021AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfercited by 178
  7. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 175
  8. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  9. 2021The Values Encoded in Machine Learning Researchcited by 172
  10. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  11. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
  12. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  13. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 142
  14. 2021Taming Visually Guided Sound Generationcited by 136
  15. 2021Real-time Deep Dynamic Characterscited by 133
  16. 2022MegaPortraits: One-shot Megapixel Neural Head Avatarscited by 131
  17. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 126
  18. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  19. 2021Pathdreamer: A World Model for Indoor Navigationcited by 118
  20. 2022Towards An End-to-End Framework for Flow-Guided Video Inpaintingcited by 113
  21. 2021Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Controlcited by 106
  22. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  23. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 106
  24. 2021FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpaintingcited by 95
  25. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
  26. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  27. 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 84
  28. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNetcited by 83
  29. 2021Driving-Signal Aware Full-Body Avatarscited by 81
  30. 2021Deepfake Video Detection Using Convolutional Vision Transformercited by 79
  31. 2021GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Drivingcited by 75
  32. 2022A Survey on Long-Tailed Visual Recognitioncited by 75
  33. 2021Enhancing Photorealism Enhancementcited by 74
  34. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  35. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69
  36. 2023OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Renderingcited by 67
  37. 2021Modeling Clothing as a Separate Layer for an Animatable Human Avatarcited by 66
  38. 2021Playable Video Generationcited by 64

50 most cited of 390 All 390 (JSON)