# Unsupervised Video-to-Video Translation

arXiv: https://arxiv.org/abs/1806.03698 · HTML: https://mlsanity.com/paper/1806.03698/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2018.

## Built on by (most cited first, 8 of 8)

- [Learning Temporally and Semantically Consistent Unpaired Video-to-video Translation Through Pseudo-Supervision From Synthetic Optical Flow](https://mlsanity.com/paper/2201.05723/) (2022)
- [ROMA: Cross-Domain Region Similarity Matching for Unpaired Nighttime Infrared to Daytime Visible Video Translation](https://mlsanity.com/paper/2204.12367/) (2022)
- [3D-Aware Multi-Class Image-to-Image Translation with NeRFs](https://mlsanity.com/paper/2303.15012/) (2023)
- [Translation-based Video-to-Video Synthesis](https://mlsanity.com/paper/2404.04283/) (2024)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches](https://mlsanity.com/paper/2310.00868/) (2023)
- [WAIT: Feature Warping for Animation to Illustration video Translation using GANs](https://mlsanity.com/paper/2310.04901/) (2023)
- [From Mannequin to Human: A Pose-Aware and Identity-Preserving Video Generation Framework for Lifelike Clothing Display](https://mlsanity.com/paper/2510.16833/) (2025)

## Abstract (arXiv)

Unsupervised image-to-image translation is a recently proposed task of translating an image to a different style or domain given only unpaired image examples at training time. In this paper, we formulate a new task of unsupervised video-to-video translation, which poses its own unique challenges. Translating video implies learning not only the appearance of objects and scenes but also realistic motion and transitions between consecutive frames.We investigate the performance of per-frame video-to-video translation using existing image-to-image translation networks, and propose a spatio-temporal 3D translator as an alternative solution to this problem. We evaluate our 3D method on multiple synthetic datasets, such as moving colorized digits, as well as the realistic segmentation-to-video GTA dataset and a new CT-to-MRI volumetric images translation dataset. Our results show that frame-wise translation produces realistic results on a single frame level but underperforms significantly on the scale of the whole video compared to our three-dimensional translation approach, which is better able to learn the complex structure of video and motion and continuity of object appearance.