# Pose-guided Generative Adversarial Net for Novel View Action Synthesis

arXiv: https://arxiv.org/abs/2110.07993 · HTML: https://mlsanity.com/paper/2110.07993/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 44 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 30 of 35)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [Temporal Generative Adversarial Nets with Singular Value Clipping](https://mlsanity.com/paper/1611.06624/) (2016)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [Pose Guided Human Video Generation](https://mlsanity.com/paper/1807.11152/) (2018)
- [Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction](https://mlsanity.com/paper/1910.02027/) (2019)
- [Adversarial Video Generation on Complex Datasets](https://mlsanity.com/paper/1907.06571/) (2019)
- [PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos](https://mlsanity.com/paper/2103.09009/) (2021)
- [Learning Dynamical Human-Joint Affinity for 3D Pose Estimation in Videos](https://mlsanity.com/paper/2109.07353/) (2021)
- [Novel Views of Objects from a Single Image](https://mlsanity.com/paper/1602.00328/) (2016)
- [Geometry-Aware Neural Rendering](https://mlsanity.com/paper/1911.04554/) (2019)

## Built on by (most cited first, 1 of 1)

- [LARNet: Latent Action Representation for Human Action Synthesis](https://mlsanity.com/paper/2110.10899/) (2021)

## Abstract (arXiv)

We focus on the problem of novel-view human action synthesis. Given an action video, the goal is to generate the same action from an unseen viewpoint. Naturally, novel view video synthesis is more challenging than image synthesis. It requires the synthesis of a sequence of realistic frames with temporal coherency. Besides, transferring the different actions to a novel target view requires awareness of action category and viewpoint change simultaneously. To address these challenges, we propose a novel framework named Pose-guided Action Separable Generative Adversarial Net (PAS-GAN), which utilizes pose to alleviate the difficulty of this task. First, we propose a recurrent pose-transformation module which transforms actions from the source view to the target view and generates novel view pose sequence in 2D coordinate space. Second, a well-transformed pose sequence enables us to separatethe action and background in the target view. We employ a novel local-global spatial transformation module to effectively generate sequential video features in the target view using these action and background features. Finally, the generated video features are used to synthesize human action with the help of a 3D decoder. Moreover, to focus on dynamic action in the video, we propose a novel multi-scale action-separable loss which further improves the video quality. We conduct extensive experiments on two large-scale multi-view human action datasets, NTU-RGBD and PKU-MMD, demonstrating the effectiveness of PAS-GAN which outperforms existing approaches.