# Open-Vocabulary Temporal Action Detection with Off-the-Shelf Image-Text Features

arXiv: https://arxiv.org/abs/2212.10596 · HTML: https://mlsanity.com/paper/2212.10596/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 53 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Speed/accuracy trade-offs for modern convolutional object detectors](https://mlsanity.com/paper/1611.10012/) (2016)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)

## Built on by (most cited first, 6 of 6)

- [Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection](https://mlsanity.com/paper/2411.10922/) (2024)
- [Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localization](https://mlsanity.com/paper/2407.07024/) (2024)
- [Open-Vocabulary Spatio-Temporal Action Detection](https://mlsanity.com/paper/2405.10832/) (2024)
- [Open-Vocabulary Action Localization with Iterative Visual Prompting](https://mlsanity.com/paper/2408.17422/) (2024)
- [One-Stage Open-Vocabulary Temporal Action Detection Leveraging Temporal Multi-scale and Action Label Features](https://mlsanity.com/paper/2404.19542/) (2024)
- [Scaling Open-Vocabulary Action Detection](https://mlsanity.com/paper/2504.03096/) (2025)

## Abstract (arXiv)

Detecting actions in untrimmed videos should not be limited to a small, closed set of classes. We present a simple, yet effective strategy for open-vocabulary temporal action detection utilizing pretrained image-text co-embeddings. Despite being trained on static images rather than videos, we show that image-text co-embeddings enable openvocabulary performance competitive with fully-supervised models. We show that the performance can be further improved by ensembling the image-text features with features encoding local motion, like optical flow based features, or other modalities, like audio. In addition, we propose a more reasonable open-vocabulary evaluation setting for the ActivityNet data set, where the category splits are based on similarity rather than random assignment.