# Translation-based Video-to-Video Synthesis

arXiv: https://arxiv.org/abs/2404.04283 · HTML: https://mlsanity.com/paper/2404.04283/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model](https://mlsanity.com/paper/1706.03458/) (2017)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [Recycle-GAN: Unsupervised Video Retargeting](https://mlsanity.com/paper/1808.05174/) (2018)
- [Deep Exemplar-based Video Colorization](https://mlsanity.com/paper/1906.09909/) (2019)
- [World-Consistent Video-to-Video Synthesis](https://mlsanity.com/paper/2007.08509/) (2020)
- [Pose Guided Human Video Generation](https://mlsanity.com/paper/1807.11152/) (2018)
- [Mocycle-GAN: Unpaired Video-to-Video Translation](https://mlsanity.com/paper/1908.09514/) (2019)
- [Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer](https://mlsanity.com/paper/1703.08516/) (2017)
- [Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data](https://mlsanity.com/paper/2103.17204/) (2021)
- [Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translation](https://mlsanity.com/paper/1908.07683/) (2019)
- [Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation](https://mlsanity.com/paper/1808.02992/) (2018)
- [I2V-GAN: Unpaired Infrared-to-Visible Video Translation](https://mlsanity.com/paper/2108.00913/) (2021)
- [Unsupervised Video-to-Video Translation](https://mlsanity.com/paper/1806.03698/) (2018)
- [Unsupervised Multimodal Video-to-Video Translation via Self-Supervised Learning](https://mlsanity.com/paper/2004.06502/) (2020)
- [Invertible Frowns: Video-to-Video Facial Emotion Translation](https://mlsanity.com/paper/2109.08061/) (2021)

## Built on by (most cited first, 1 of 1)

- [Video Motion Transfer with Diffusion Transformers](https://mlsanity.com/paper/2412.07776/) (2024)

## Abstract (arXiv)

Translation-based Video Synthesis (TVS) has emerged as a vital research area in computer vision, aiming to facilitate the transformation of videos between distinct domains while preserving both temporal continuity and underlying content features. This technique has found wide-ranging applications, encompassing video super-resolution, colorization, segmentation, and more, by extending the capabilities of traditional image-to-image translation to the temporal domain. One of the principal challenges faced in TVS is the inherent risk of introducing flickering artifacts and inconsistencies between frames during the synthesis process. This is particularly challenging due to the necessity of ensuring smooth and coherent transitions between video frames. Efforts to tackle this challenge have induced the creation of diverse strategies and algorithms aimed at mitigating these unwanted consequences. This comprehensive review extensively examines the latest progress in the realm of TVS. It thoroughly investigates emerging methodologies, shedding light on the fundamental concepts and mechanisms utilized for proficient video synthesis. This survey also illuminates their inherent strengths, limitations, appropriate applications, and potential avenues for future development.