# Physical Context and Timing Aware Sequence Generating GANs

arXiv: https://arxiv.org/abs/2110.04077 · HTML: https://mlsanity.com/paper/2110.04077/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Video Frame Interpolation via Adaptive Convolution](https://mlsanity.com/paper/1703.07514/) (2017)
- [Enforcing Statistical Constraints in Generative Adversarial Networks for Modeling Chaotic Dynamical Systems](https://mlsanity.com/paper/1905.06841/) (2019)
- [Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities](https://mlsanity.com/paper/1701.06264/) (2017)
- [Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images](https://mlsanity.com/paper/1803.03435/) (2018)
- [Stochastic Dynamics for Video Infilling](https://mlsanity.com/paper/1809.00263/) (2018)
- [cGANs with Multi-Hinge Loss](https://mlsanity.com/paper/1912.04216/) (2019)
- [Long-Term Video Interpolation with Bidirectional Predictive Network](https://mlsanity.com/paper/1706.03947/) (2017)
- [From Here to There: Video Inbetweening Using Direct 3D Convolutions](https://mlsanity.com/paper/1905.10240/) (2019)

## Abstract (arXiv)

Generative Adversarial Networks (GANs) have shown remarkable successes in generating realistic images and interpolating changes between images. Existing models, however, do not take into account physical contexts behind images in generating the images, which may cause unrealistic changes. Furthermore, it is difficult to generate the changes at a specific timing and they often do not match with actual changes. This paper proposes a novel GAN, named Physical Context and Timing aware sequence generating GANs (PCTGAN), that generates an image in a sequence at a specific timing between two images with considering physical contexts behind them. Our method consists of three components: an encoder, a generator, and a discriminator. The encoder estimates latent vectors from the beginning and ending images, their timings, and a target timing. The generator generates images and the physical contexts at the beginning, ending, and target timing from the corresponding latent vectors. The discriminator discriminates whether the generated images and contexts are real or not. In the experiments, PCTGAN is applied to a data set of sequential changes of shapes in die forging processes. We show that both timing and physical contexts are effective in generating sequential images.