# Jointly Trained Image and Video Generation using Residual Vectors

arXiv: https://arxiv.org/abs/1912.07991 · HTML: https://mlsanity.com/paper/1912.07991/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2019.

## Built on by (most cited first, 5 of 5)

- [GenTron: Diffusion Transformers for Image and Video Generation](https://mlsanity.com/paper/2312.04557/) (2023)
- [Vlogger: Make Your Dream A Vlog](https://mlsanity.com/paper/2401.09414/) (2024)
- [Hamiltonian latent operators for content and motion disentanglement in image sequences](https://mlsanity.com/paper/2112.01641/) (2021)
- [NeurInt : Learning to Interpolate through Neural ODEs](https://mlsanity.com/paper/2111.04123/) (2021)
- [Axial Neural Networks for Dimension-Free Foundation Models](https://mlsanity.com/paper/2510.13665/) (2025)

## Abstract (arXiv)

In this work, we propose a modeling technique for jointly training image and video generation models by simultaneously learning to map latent variables with a fixed prior onto real images and interpolate over images to generate videos. The proposed approach models the variations in representations using residual vectors encoding the change at each time step over a summary vector for the entire video. We utilize the technique to jointly train an image generation model with a fixed prior along with a video generation model lacking constraints such as disentanglement. The joint training enables the image generator to exploit temporal information while the video generation model learns to flexibly share information across frames. Moreover, experimental results verify our approach's compatibility with pre-training on videos or images and training on datasets containing a mixture of both. A comprehensive set of quantitative and qualitative evaluations reveal the improvements in sample quality and diversity over both video generation and image generation baselines. We further demonstrate the technique's capabilities of exploiting similarity in features across frames by applying it to a model based on decomposing the video into motion and content. The proposed model allows minor variations in content across frames while maintaining the temporal dependence through latent vectors encoding the pose or motion features.