# Few-Shot Transformation of Common Actions into Time and Space

arXiv: https://arxiv.org/abs/2104.02439 · HTML: https://mlsanity.com/paper/2104.02439/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 6 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector](https://mlsanity.com/paper/1908.01998/) (2019)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [Online Real-time Multiple Spatiotemporal Action Localisation and Prediction](https://mlsanity.com/paper/1611.08563/) (2016)
- [STEP: Spatio-Temporal Progressive Learning for Video Action Detection](https://mlsanity.com/paper/1904.09288/) (2019)
- [Learning Spatiotemporal Features with 3D Convolutional Networks](https://mlsanity.com/paper/1412.0767/) (2014)
- [Objects2action: Classifying and localizing actions without any video example](https://mlsanity.com/paper/1510.06939/) (2015)
- [Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions](https://mlsanity.com/paper/1707.09145/) (2017)
- [Localizing the Common Action Among a Few Videos](https://mlsanity.com/paper/2008.05826/) (2020)
- [Spatio-temporal Video Re-localization by Warp LSTM](https://mlsanity.com/paper/1905.03922/) (2019)
- [Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs](https://mlsanity.com/paper/1601.02129/) (2016)

## Abstract (arXiv)

This paper introduces the task of few-shot common action localization in time and space. Given a few trimmed support videos containing the same but unknown action, we strive for spatio-temporal localization of that action in a long untrimmed query video. We do not require any class labels, interval bounds, or bounding boxes. To address this challenging task, we introduce a novel few-shot transformer architecture with a dedicated encoder-decoder structure optimized for joint commonality learning and localization prediction, without the need for proposals. Experiments on our reorganizations of the AVA and UCF101-24 datasets show the effectiveness of our approach for few-shot common action localization, even when the support videos are noisy. Although we are not specifically designed for common localization in time only, we also compare favorably against the few-shot and one-shot state-of-the-art in this setting. Lastly, we demonstrate that the few-shot transformer is easily extended to common action localization per pixel.