# Training-Free Zero-Shot Temporal Action Detection with Vision-Language Models

arXiv: https://arxiv.org/abs/2501.13795 · HTML: https://mlsanity.com/paper/2501.13795/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 54 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Detecting and Preventing Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2308.06394/) (2023)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Learning in the Frequency Domain](https://mlsanity.com/paper/2002.12416/) (2020)
- [TriDet: Temporal Action Detection with Relative Boundary Modeling](https://mlsanity.com/paper/2303.07347/) (2023)
- [Single Shot Temporal Action Detection](https://mlsanity.com/paper/1710.06236/) (2017)
- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)
- [Temporal Context Aggregation Network for Temporal Action Proposal Refinement](https://mlsanity.com/paper/2103.13141/) (2021)
- [Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding](https://mlsanity.com/paper/2403.09626/) (2024)
- [Video Self-Stitching Graph Network for Temporal Action Localization](https://mlsanity.com/paper/2011.14598/) (2020)
- [Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting](https://mlsanity.com/paper/2304.03307/) (2023)
- [AutoLoc: Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/1807.08333/) (2018)
- [Diagnosing Error in Temporal Action Detectors](https://mlsanity.com/paper/1807.10706/) (2018)
- [Zero-Shot Temporal Action Detection via Vision-Language Prompting](https://mlsanity.com/paper/2207.08184/) (2022)
- [End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames](https://mlsanity.com/paper/2311.17241/) (2023)
- [Video Test-Time Adaptation for Action Recognition](https://mlsanity.com/paper/2211.15393/) (2022)
- [BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detection](https://mlsanity.com/paper/2205.02717/) (2022)

## Built on by (most cited first, 4 of 4)

- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [Bitstream Action Recognition is Byte Modeling](https://mlsanity.com/paper/2608.15695/) (2026)
- [High-Speed Vision Improves Zero-Shot Semantic Understanding of Human Actions](https://mlsanity.com/paper/2605.00496/) (2026)
- [VidParse: Online Parsing of Egocentric Procedures Like a Pro](https://mlsanity.com/paper/2608.27562/) (2026)

## Abstract (arXiv)

Existing zero-shot temporal action detection (ZSTAD) methods predominantly use fully supervised or unsupervised strategies to recognize unseen activities. However, these training-based methods are prone to domain shifts and require high computational costs, which hinder their practical applicability in real-world scenarios. In this paper, unlike previous works, we propose a training-Free Zero-shot temporal Action Detection (FreeZAD) method, leveraging existing vision-language (ViL) models to directly classify and localize unseen activities within untrimmed videos without any additional fine-tuning or adaptation. We mitigate the need for explicit temporal modeling and reliance on pseudo-label quality by designing the LOGarithmic decay weighted Outer-Inner-Contrastive Score (LogOIC) and frequency-based Actionness Calibration. Furthermore, we introduce a test-time adaptation (TTA) strategy using Prototype-Centric Sampling (PCS) to expand FreeZAD, enabling ViL models to adapt more effectively for ZSTAD. Extensive experiments on the THUMOS14 and ActivityNet-1.3 datasets demonstrate that our training-free method outperforms state-of-the-art unsupervised methods while requiring only 1/13 of the runtime. When equipped with TTA, the enhanced method further narrows the gap with fully supervised methods.