# Unsupervised Multimodal Video-to-Video Translation via Self-Supervised Learning

arXiv: https://arxiv.org/abs/2004.06502 · HTML: https://mlsanity.com/paper/2004.06502/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2020.

## Built on by (most cited first, 7 of 7)

- [I2V-GAN: Unpaired Infrared-to-Visible Video Translation](https://mlsanity.com/paper/2108.00913/) (2021)
- [Learning Temporally and Semantically Consistent Unpaired Video-to-video Translation Through Pseudo-Supervision From Synthetic Optical Flow](https://mlsanity.com/paper/2201.05723/) (2022)
- [3D-Aware Multi-Class Image-to-Image Translation with NeRFs](https://mlsanity.com/paper/2303.15012/) (2023)
- [Translation-based Video-to-Video Synthesis](https://mlsanity.com/paper/2404.04283/) (2024)
- [From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs)](https://mlsanity.com/paper/2601.05059/) (2026)
- [WAIT: Feature Warping for Animation to Illustration video Translation using GANs](https://mlsanity.com/paper/2310.04901/) (2023)
- [SVS-GAN: Leveraging GANs for Semantic Video Synthesis](https://mlsanity.com/paper/2409.06074/) (2024)

## Abstract (arXiv)

Existing unsupervised video-to-video translation methods fail to produce translated videos which are frame-wise realistic, semantic information preserving and video-level consistent. In this work, we propose UVIT, a novel unsupervised video-to-video translation model. Our model decomposes the style and the content, uses the specialized encoder-decoder structure and propagates the inter-frame information through bidirectional recurrent neural network (RNN) units. The style-content decomposition mechanism enables us to achieve style consistent video translation results as well as provides us with a good interface for modality flexible translation. In addition, by changing the input frames and style codes incorporated in our translation, we propose a video interpolation loss, which captures temporal information within the sequence to train our building blocks in a self-supervised manner. Our model can produce photo-realistic, spatio-temporal consistent translated videos in a multimodal way. Subjective and objective experimental results validate the superiority of our model over existing methods. More details can be found on our project website: https://uvit.netlify.com