mlsanityData to 14 Oct 2026

Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation

arXiv 2602.21406 ↗First posted 24 Feb 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Temporal Action Segmentation (TAS) requires dividing videos into action segments, yet the vast space of activities and alternative breakdowns makes collecting comprehensive datasets infeasible. Existing methods remain limited to closed vocabularies and fixed label sets. In this work, we explore the largely unexplored problem of Open-Vocabulary Zero-Shot Temporal Action Segmentation (OVTAS) by leveraging the strong zero-shot capabilities of Vision-Language Models (VLMs). We introduce a training-free pipeline that follows a segmentation-by-classification design: Frame-Action Embedding Similarity (FAES) matches video frames to candidate action labels, and Similarity-Matrix Temporal Segmentation (SMTS) enforces temporal consistency. Beyond proposing OVTAS, we present a systematic study across 14 diverse VLMs, providing the first broad analysis of their suitability for open-vocabulary action segmentation. Experiments on standard benchmarks show that OVTAS achieves strong results without task-specific supervision, underscoring the potential of VLMs for structured temporal understanding.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 35 references that are in the map. First posted on arXiv 24 Feb 2026.

Builds on

24 of 35 references.

References · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  4. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  5. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 87
  6. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  7. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  8. 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
  9. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  10. 2021Temporal Action Segmentation from Timestamp Supervisioncited by 48
  11. 2016Weakly supervised learning of actions from transcriptscited by 46
  12. 2024Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentationcited by 38
12 more
  1. 2020Set-Constrained Viterbi for Set-Supervised Action Segmentationcited by 35
  2. 2021Action Shuffle Alternating Learning for Unsupervised Action Segmentationcited by 33
  3. 2021Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activitiescited by 26
  4. 2024Hierarchical Vector Quantization for Unsupervised Action Segmentationcited by 19
  5. 2022Timestamp-Supervised Action Segmentation with Graph Convolutional Networkscited by 18
  6. 2022Leveraging Action Affinity and Continuity for Semi-supervised Temporal Action Segmentationcited by 17
  7. 2021SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentationcited by 15
  8. 2024Hierarchical NeuroSymbolic Approach for Comprehensive and Explainable Action Quality Assessmentcited by 14
  9. 2019Fine-grained Action Segmentation using the Semi-Supervised Action GANcited by 12
  10. 2023Leveraging triplet loss for unsupervised action segmentationcited by 10
  11. 2024Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIPcited by 9
  12. 2024A Backbone for Long-Horizon Robot Task Understandingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detectioncited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Spatio-Temporal Zero-shot Action Recognition with Language-driven Description Attributes
  2. 2023ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection
  3. 2026Novel Semantic Prompting for Zero-Shot Action Recognition
  4. 2026Action100M: A Large-scale Video Action Dataset
  5. 2025Training-Free Zero-Shot Temporal Action Detection with Vision-Language Models
  6. 2024Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection
  7. 2024Text-Enhanced Zero-Shot Action Recognition: A training-free approach
  8. 2026Zero-Shot Temporal Action Localization Through Textual Guidance