# Towards Universal Representation for Unseen Action Recognition

arXiv: https://arxiv.org/abs/1803.08460 · HTML: https://mlsanity.com/paper/1803.08460/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2018.

## Built on by (most cited first, 30 of 44)

- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [Fine-tuned CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2212.03640/) (2022)
- [Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting](https://mlsanity.com/paper/2304.03307/) (2023)
- [Elaborative Rehearsal for Zero-shot Action Recognition](https://mlsanity.com/paper/2108.02833/) (2021)
- [VicTR: Video-conditioned Text Representations for Activity Recognition](https://mlsanity.com/paper/2304.02560/) (2023)
- [Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language](https://mlsanity.com/paper/2203.03598/) (2022)
- [Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization](https://mlsanity.com/paper/2308.03950/) (2023)
- [Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models](https://mlsanity.com/paper/2207.07646/) (2022)
- [OmniVid: A Generative Framework for Universal Video Understanding](https://mlsanity.com/paper/2403.17935/) (2024)
- [Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learning](https://mlsanity.com/paper/2308.04828/) (2023)
- [Object Priors for Classifying and Localizing Unseen Actions](https://mlsanity.com/paper/2104.04715/) (2021)
- [Cross-modal Representation Learning for Zero-shot Action Recognition](https://mlsanity.com/paper/2205.01657/) (2022)
- [Deep Unsupervised Key Frame Extraction for Efficient Video Classification](https://mlsanity.com/paper/2211.06742/) (2022)
- [TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale](https://mlsanity.com/paper/2305.14173/) (2023)
- [EZ-CLIP: Efficient Zeroshot Video Action Recognition](https://mlsanity.com/paper/2312.08010/) (2023)
- [CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition](https://mlsanity.com/paper/2101.07042/) (2021)
- [Video Action Recognition with Attentive Semantic Units](https://mlsanity.com/paper/2303.09756/) (2023)
- [Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition](https://mlsanity.com/paper/2311.15619/) (2023)
- [Alignment-Uniformity aware Representation Learning for Zero-shot Video Classification](https://mlsanity.com/paper/2203.15381/) (2022)
- [Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions](https://mlsanity.com/paper/2106.03158/) (2021)
- [Continuous Human Action Recognition for Human-Machine Interaction: A Review](https://mlsanity.com/paper/2202.13096/) (2022)
- [Language-based Action Concept Spaces Improve Video Self-Supervised Learning](https://mlsanity.com/paper/2307.10922/) (2023)
- [Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignment](https://mlsanity.com/paper/2409.14336/) (2024)
- [Zero-Shot Action Recognition from Diverse Object-Scene Compositions](https://mlsanity.com/paper/2110.13479/) (2021)
- [Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognition](https://mlsanity.com/paper/2312.02226/) (2023)
- [ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognition](https://mlsanity.com/paper/2401.11654/) (2024)
- [Discriminative Latent Semantic Graph for Video Captioning](https://mlsanity.com/paper/2108.03662/) (2021)
- [Tell me what you see: A zero-shot action recognition method based on natural language descriptions](https://mlsanity.com/paper/2112.09976/) (2021)
- [Videoprompter: an ensemble of foundational models for zero-shot video understanding](https://mlsanity.com/paper/2310.15324/) (2023)
- [Continual Learning Improves Zero-Shot Action Recognition](https://mlsanity.com/paper/2410.10497/) (2024)

## Abstract (arXiv)

Unseen Action Recognition (UAR) aims to recognise novel action categories without training examples. While previous methods focus on inner-dataset seen/unseen splits, this paper proposes a pipeline using a large-scale training source to achieve a Universal Representation (UR) that can generalise to a more realistic Cross-Dataset UAR (CD-UAR) scenario. We first address UAR as a Generalised Multiple-Instance Learning (GMIL) problem and discover 'building-blocks' from the large-scale ActivityNet dataset using distribution kernels. Essential visual and semantic components are preserved in a shared space to achieve the UR that can efficiently generalise to new datasets. Predicted UR exemplars can be improved by a simple semantic adaptation, and then an unseen action can be directly recognised using UR during the test. Without further training, extensive experiments manifest significant improvements over the UCF101 and HMDB51 benchmarks.