# Action2Vec: A Crossmodal Embedding Approach to Action Learning

arXiv: https://arxiv.org/abs/1901.00484 · HTML: https://mlsanity.com/paper/1901.00484/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2019.

## Built on by (most cited first, 19 of 19)

- [Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language](https://mlsanity.com/paper/2203.03598/) (2022)
- [Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models](https://mlsanity.com/paper/2207.07646/) (2022)
- [One-shot action recognition in challenging therapy scenarios](https://mlsanity.com/paper/2102.08997/) (2021)
- [Cross-modal Representation Learning for Zero-shot Action Recognition](https://mlsanity.com/paper/2205.01657/) (2022)
- [Towards Zero-shot Sign Language Recognition](https://mlsanity.com/paper/2201.05914/) (2022)
- [Alignment-Uniformity aware Representation Learning for Zero-shot Video Classification](https://mlsanity.com/paper/2203.15381/) (2022)
- [Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions](https://mlsanity.com/paper/2106.03158/) (2021)
- [ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognition](https://mlsanity.com/paper/2401.11654/) (2024)
- [Domain Adaptation in Multi-View Embedding for Cross-Modal Video Retrieval](https://mlsanity.com/paper/2110.12812/) (2021)
- [Domain and View-point Agnostic Hand Action Recognition](https://mlsanity.com/paper/2103.02303/) (2021)
- [Pretraining on Interactions for Learning Grounded Affordance Representations](https://mlsanity.com/paper/2207.02272/) (2022)
- [ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos](https://mlsanity.com/paper/2108.10059/) (2021)
- [Universal Prototype Transport for Zero-Shot Action Recognition and Localization](https://mlsanity.com/paper/2203.03971/) (2022)
- [Multi-Modal Zero-Shot Sign Language Recognition](https://mlsanity.com/paper/2109.00796/) (2021)
- [TNT: Text-Conditioned Network with Transductive Inference for Few-Shot Video Classification](https://mlsanity.com/paper/2106.11173/) (2021)
- [End-to-End Semantic Video Transformer for Zero-Shot Action Recognition](https://mlsanity.com/paper/2203.05156/) (2022)
- [Video-adverb retrieval with compositional adverb-action embeddings](https://mlsanity.com/paper/2309.15086/) (2023)
- [Few-Shot Action Localization without Knowing Boundaries](https://mlsanity.com/paper/2106.04150/) (2021)
- [Improving Zero-Shot Action Recognition using Human Instruction with Text Description](https://mlsanity.com/paper/2301.08874/) (2023)

## Abstract (arXiv)

We describe a novel cross-modal embedding space for actions, named Action2Vec, which combines linguistic cues from class labels with spatio-temporal features derived from video clips. Our approach uses a hierarchical recurrent network to capture the temporal structure of video features. We train our embedding using a joint loss that combines classification accuracy with similarity to Word2Vec semantics. We evaluate Action2Vec by performing zero shot action recognition and obtain state of the art results on three standard datasets. In addition, we present two novel analogy tests which quantify the extent to which our joint embedding captures distributional semantics. This is the first joint embedding space to combine verbs and action videos, and the first to be thoroughly evaluated with respect to its distributional semantics.