# GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition

arXiv: https://arxiv.org/abs/2105.11789 · HTML: https://mlsanity.com/paper/2105.11789/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 25 May 2021.

## Builds on (most cited first, 25 of 25)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [C2AE: Class Conditioned Auto-Encoder for Open-set Recognition](https://mlsanity.com/paper/1904.01198/) (2019)
- [Zero-Shot Learning by Convex Combination of Semantic Embeddings](https://mlsanity.com/paper/1312.5650/) (2013)
- [Multi-modal Cycle-consistent Generalized Zero-Shot Learning](https://mlsanity.com/paper/1808.00136/) (2018)
- [Out-of-Distribution Detection for Generalized Zero-Shot Action Recognition](https://mlsanity.com/paper/1904.08703/) (2019)
- [Learning Attributes Equals Multi-Source Domain Generalization](https://mlsanity.com/paper/1605.00743/) (2016)
- [Towards Universal Representation for Unseen Action Recognition](https://mlsanity.com/paper/1803.08460/) (2018)
- [Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization](https://mlsanity.com/paper/1904.06836/) (2019)
- [A Generative Approach to Zero-Shot and Few-Shot Action Recognition](https://mlsanity.com/paper/1801.09086/) (2018)
- [Transductive Zero-Shot Action Recognition by Word-Vector Embedding](https://mlsanity.com/paper/1511.04458/) (2015)
- [Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning](https://mlsanity.com/paper/1605.08151/) (2016)
- [Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation](https://mlsanity.com/paper/1611.08663/) (2016)
- [Objects2action: Classifying and localizing actions without any video example](https://mlsanity.com/paper/1510.06939/) (2015)
- [Semantic Embedding Space for Zero-Shot Action Recognition](https://mlsanity.com/paper/1502.01540/) (2015)
- [Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions](https://mlsanity.com/paper/1707.09145/) (2017)
- [Visual Data Synthesis via GAN for Zero-Shot Video Classification](https://mlsanity.com/paper/1804.10073/) (2018)
- [Order-aware Convolutional Pooling for Video Based Action Recognition](https://mlsanity.com/paper/1602.00224/) (2016)

## Built on by (most cited first, 2 of 2)

- [Tell me what you see: A zero-shot action recognition method based on natural language descriptions](https://mlsanity.com/paper/2112.09976/) (2021)
- [Mutual Balancing in State-Object Components for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2211.10647/) (2022)

## Abstract (arXiv)

Zero-shot action recognition can recognize samples of unseen classes that are unavailable in training by exploring common latent semantic representation in samples. However, most methods neglected the connotative relation and extensional relation between the action classes, which leads to the poor generalization ability of the zero-shot learning. Furthermore, the learned classifier incline to predict the samples of seen class, which leads to poor classification performance. To solve the above problems, we propose a two-stage deep neural network for zero-shot action recognition, which consists of a feature generation sub-network serving as the sampling stage and a graph attention sub-network serving as the classification stage. In the sampling stage, we utilize a generative adversarial networks (GAN) trained by action features and word vectors of seen classes to synthesize the action features of unseen classes, which can balance the training sample data of seen classes and unseen classes. In the classification stage, we construct a knowledge graph (KG) based on the relationship between word vectors of action classes and related objects, and propose a graph convolution network (GCN) based on attention mechanism, which dynamically updates the relationship between action classes and objects, and enhances the generalization ability of zero-shot learning. In both stages, we all use word vectors as bridges for feature generation and classifier generalization from seen classes to unseen classes. We compare our method with state-of-the-art methods on UCF101 and HMDB51 datasets. Experimental results show that our proposed method improves the classification performance of the trained classifier and achieves higher accuracy.