# Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation

arXiv: https://arxiv.org/abs/1811.09393 · HTML: https://mlsanity.com/paper/1811.09393/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2018.

## Built on by (most cited first, 18 of 18)

- [MoDec-GS: Global-to-Local Motion Decomposition and Temporal Interval Adjustment for Compact Dynamic 3D Gaussian Splatting](https://mlsanity.com/paper/2501.03714/) (2025)
- [Blind Video Deflickering by Neural Filtering with a Flawed Atlas](https://mlsanity.com/paper/2303.08120/) (2023)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [Triple-level Model Inferred Collaborative Network Architecture for Video Deraining](https://mlsanity.com/paper/2111.04459/) (2021)
- [Neural Network Libraries: A Deep Learning Framework Designed from Engineers' Perspectives](https://mlsanity.com/paper/2102.06725/) (2021)
- [Real-Time Super-Resolution System of 4K-Video Based on Deep Learning](https://mlsanity.com/paper/2107.05307/) (2021)
- [A Survey on Deep learning based Document Image Enhancement](https://mlsanity.com/paper/2112.02719/) (2021)
- [MoRF: Mobile Realistic Fullbody Avatars from a Monocular Video](https://mlsanity.com/paper/2303.10275/) (2023)
- [Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements](https://mlsanity.com/paper/2312.07835/) (2023)
- [Continuous Space-Time Video Super-Resolution with 3D Fourier Fields](https://mlsanity.com/paper/2509.26325/) (2025)
- [Task Agnostic Restoration of Natural Video Dynamics](https://mlsanity.com/paper/2206.03753/) (2022)
- [SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control](https://mlsanity.com/paper/2605.27891/) (2026)
- [MoRel: Long-Range Flicker-Free 4D Motion Modeling via Anchor Relay-based Bidirectional Blending with Hierarchical Densification](https://mlsanity.com/paper/2512.09270/) (2025)
- [Mango-GS: Enhancing Spatio-Temporal Consistency in Dynamic Scenes Reconstruction using Multi-Frame Node-Guided 4D Gaussian Splatting](https://mlsanity.com/paper/2603.11543/) (2026)
- [A comparative study of various Deep Learning techniques for spatio-temporal Super-Resolution reconstruction of Forced Isotropic Turbulent flows](https://mlsanity.com/paper/2107.03361/) (2021)
- [RBPGAN: Recurrent Back-Projection GAN for Video Super Resolution](https://mlsanity.com/paper/2311.09178/) (2023)
- [Neural-Network-Enhanced Metalens Camera for High-Definition, Dynamic Imaging in the Long-Wave Infrared Spectrum](https://mlsanity.com/paper/2411.17139/) (2024)
- [Improving Temporal Consistency and Fidelity at Inference-time in Perceptual Video Restoration by Zero-shot Image-based Diffusion Models](https://mlsanity.com/paper/2510.25420/) (2025)

## Abstract (arXiv)

Our work explores temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for a variety of areas, temporal relationships in the generated data are much less explored. Natural temporal changes are crucial for sequential generation tasks, e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as L2 over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail. For unpaired video translation, existing approaches modify the generator networks to form spatio-temporal cycle consistencies. In contrast, we focus on improving learning objectives and propose a temporally self-supervised algorithm. For both tasks, we show that temporal adversarial learning is key to achieving temporally coherent solutions without sacrificing spatial detail. We also propose a novel Ping-Pong loss to improve the long-term temporal consistency. It effectively prevents recurrent networks from accumulating artifacts temporally without depressing detailed features. Additionally, we propose a first set of metrics to quantitatively evaluate the accuracy as well as the perceptual quality of the temporal evolution. A series of user studies confirm the rankings computed with these metrics. Code, data, models, and results are provided at https://github.com/thunil/TecoGAN. The project page https://ge.in.tum.de/publications/2019-tecogan-chu/ contains supplemental materials.