# Hierarchical Video Generation from Orthogonal Information: Optical Flow and Texture

arXiv: https://arxiv.org/abs/1711.09618 · HTML: https://mlsanity.com/paper/1711.09618/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2017.

## Built on by (most cited first, 16 of 16)

- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [Latent Image Animator: Learning to Animate Images via Latent Space Navigation](https://mlsanity.com/paper/2203.09043/) (2022)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [Synthetic Data -- what, why and how?](https://mlsanity.com/paper/2205.03257/) (2022)
- [LEO: Generative Latent Image Animator for Human Video Synthesis](https://mlsanity.com/paper/2305.03989/) (2023)
- [InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation](https://mlsanity.com/paper/2101.03049/) (2021)
- [ObjCtrl-2.5D: Training-free Object Control with Camera Poses](https://mlsanity.com/paper/2412.07721/) (2024)
- [A review of Generative Adversarial Networks (GANs) and its applications in a wide variety of disciplines -- From Medical to Remote Sensing](https://mlsanity.com/paper/2110.01442/) (2021)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [Text-driven Video Prediction](https://mlsanity.com/paper/2210.02872/) (2022)
- [Versatile Face Animator: Driving Arbitrary 3D Facial Avatar in RGBD Space](https://mlsanity.com/paper/2308.06076/) (2023)
- [LIA-X: Interpretable Latent Portrait Animator](https://mlsanity.com/paper/2508.09959/) (2025)
- [Efficient training for future video generation based on hierarchical disentangled representation of latent variables](https://mlsanity.com/paper/2106.03502/) (2021)
- [LARNet: Latent Action Representation for Human Action Synthesis](https://mlsanity.com/paper/2110.10899/) (2021)
- [Autoencoding Video Latents for Adversarial Video Generation](https://mlsanity.com/paper/2201.06888/) (2022)
- [MeVGAN: GAN-based Plugin Model for Video Generation with Applications in Colonoscopy](https://mlsanity.com/paper/2311.03884/) (2023)

## Abstract (arXiv)

Learning to represent and generate videos from unlabeled data is a very challenging problem. To generate realistic videos, it is important not only to ensure that the appearance of each frame is real, but also to ensure the plausibility of a video motion and consistency of a video appearance in the time direction. The process of video generation should be divided according to these intrinsic difficulties. In this study, we focus on the motion and appearance information as two important orthogonal components of a video, and propose Flow-and-Texture-Generative Adversarial Networks (FTGAN) consisting of FlowGAN and TextureGAN. In order to avoid a huge annotation cost, we have to explore a way to learn from unlabeled data. Thus, we employ optical flow as motion information to generate videos. FlowGAN generates optical flow, which contains only the edge and motion of the videos to be begerated. On the other hand, TextureGAN specializes in giving a texture to optical flow generated by FlowGAN. This hierarchical approach brings more realistic videos with plausible motion and appearance consistency. Our experiments show that our model generates more plausible motion videos and also achieves significantly improved performance for unsupervised action classification in comparison to previous GAN works. In addition, because our model generates videos from two independent information, our model can generate new combinations of motion and attribute that are not seen in training data, such as a video in which a person is doing sit-up in a baseball ground.