mlsanityData to 7 Oct 2026

Open-Vocabulary Temporal Action Detection with Off-the-Shelf Image-Text Features

arXiv 2212.10596 ↗First posted 20 Dec 2022cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Detecting actions in untrimmed videos should not be limited to a small, closed set of classes. We present a simple, yet effective strategy for open-vocabulary temporal action detection utilizing pretrained image-text co-embeddings. Despite being trained on static images rather than videos, we show that image-text co-embeddings enable openvocabulary performance competitive with fully-supervised models. We show that the performance can be further improved by ensembling the image-text features with features encoding local motion, like optical flow based features, or other modalities, like audio. In addition, we propose a more reasonable open-vocabulary evaluation setting for the ActivityNet data set, where the category splits are based on similarity rather than random assignment.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

43 of 53 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2016Layer Normalizationcited by 5,579
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  10. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  11. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  12. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
31 more
  1. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  2. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  3. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  4. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  5. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  6. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  7. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  8. 2017Temporal Action Detection with Structured Segment Networkscited by 265
  9. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 248
  10. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  11. 2019Graph Convolutional Networks for Temporal Action Localizationcited by 208
  12. 2016Speed/accuracy trade-offs for modern convolutional object detectorscited by 198
  13. 2018Rethinking the Faster R-CNN Architecture for Temporal Action Localizationcited by 194
  14. 2018Zero-Shot Object Detectioncited by 189
  15. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  16. 2017R-C3D: Region Convolutional 3D Network for Temporal Activity Detectioncited by 161
  17. 2021Learning Salient Boundary Feature for Anchor-free Temporal Action Localizationcited by 156
  18. 2021End-to-end Temporal Action Detection with Transformercited by 150
  19. 2017UntrimmedNets for Weakly Supervised Action Recognition and Detectioncited by 129
  20. 2017CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videoscited by 112
  21. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  22. 2020TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Taskscited by 77
  23. 2020Revisiting Anchor Mechanisms for Temporal Action Localizationcited by 75
  24. 2018Diagnosing Error in Temporal Action Detectorscited by 71
  25. 2019Dont Even Look Once: Synthesizing Features for Zero-Shot Detectioncited by 58
  26. 2016CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016cited by 53
  27. 2020Learning Video Representations from Textual Web Supervisioncited by 39
  28. 2022RCL: Recurrent Continuous Localization for Temporal Action Detectioncited by 25
  29. 2020D2-Net: Weakly-Supervised Action Localization via Discriminative Embeddings and Denoised Activationscited by 23
  30. 2022OpenTAL: Towards Open Set Temporal Action Localizationcited by 23
  31. 2020Weakly Supervised Temporal Action Localization Using Deep Metric Learningcited by 14

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 1.

6 papers.

  1. 2024Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detectioncited by 8
  2. 2024Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localizationcited by 6
  3. 2024Open-Vocabulary Spatio-Temporal Action Detectioncited by 5
  4. 2024Open-Vocabulary Action Localization with Iterative Visual Promptingcited by 5
  5. 2024One-Stage Open-Vocabulary Temporal Action Detection Leveraging Temporal Multi-scale and Action Label Featurescited by 2
  6. 2025Scaling Open-Vocabulary Action Detectioncited by 0