# Prompt-Guided Zero-Shot Anomaly Action Recognition using Pretrained Deep Skeleton Features

arXiv: https://arxiv.org/abs/2303.15167 · HTML: https://mlsanity.com/paper/2303.15167/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 48 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2010.05903/) (2020)
- [Modeling the Distribution of Normal Data in Pre-Trained Deep Features for Anomaly Detection](https://mlsanity.com/paper/2005.14140/) (2020)
- [Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos](https://mlsanity.com/paper/1903.03295/) (2019)
- [Generative Cooperative Learning for Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2203.03962/) (2022)
- [Graph Embedded Pose Clustering for Anomaly Detection](https://mlsanity.com/paper/1912.11850/) (2019)
- [Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation](https://mlsanity.com/paper/2203.10593/) (2022)
- [Zero-Shot Temporal Action Detection via Vision-Language Prompting](https://mlsanity.com/paper/2207.08184/) (2022)
- [GODS: Generalized One-class Discriminative Subspaces for Anomaly Detection](https://mlsanity.com/paper/1908.05884/) (2019)
- [Pelee: A Real-Time Object Detection System on Mobile Devices](https://mlsanity.com/paper/1804.06882/) (2018)

## Built on by (most cited first, 21 of 21)

- [Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2502.07601/) (2025)
- [Deep Graph Anomaly Detection: A Survey and New Perspectives](https://mlsanity.com/paper/2409.09957/) (2024)
- [OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding](https://mlsanity.com/paper/2406.07471/) (2024)
- [MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objects](https://mlsanity.com/paper/2412.04867/) (2024)
- [Fine-Grained Side Information Guided Dual-Prompts for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2404.07487/) (2024)
- [Random Word Data Augmentation with CLIP for Zero-Shot Anomaly Detection](https://mlsanity.com/paper/2308.11119/) (2023)
- [Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2411.11288/) (2024)
- [Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learning](https://mlsanity.com/paper/2405.20606/) (2024)
- [Privacy-Preserving Video Anomaly Detection: A Survey](https://mlsanity.com/paper/2411.14565/) (2024)
- [Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomaly](https://mlsanity.com/paper/2412.07183/) (2024)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [Vision-Language Models Assisted Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2409.14109/) (2024)
- [Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection](https://mlsanity.com/paper/2412.03044/) (2024)
- [Unveiling Context-Related Anomalies: Knowledge Graph Empowered Decoupling of Scene and Action for Human-Related Video Anomaly Detection](https://mlsanity.com/paper/2409.03236/) (2024)
- [Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2511.09388/) (2025)
- [Learn Suspected Anomalies from Event Prompts for Video Anomaly Detection](https://mlsanity.com/paper/2403.01169/) (2024)
- [Pose-Aware Weakly-Supervised Action Segmentation](https://mlsanity.com/paper/2504.05700/) (2025)
- [Emotion Recognition from Skeleton Data: A Comprehensive Survey](https://mlsanity.com/paper/2507.18026/) (2025)
- [Action Hints: Semantic Typicality and Context Uniqueness for Generalizable Skeleton-based Video Anomaly Detection](https://mlsanity.com/paper/2509.11058/) (2025)
- [SkeletonAgent: An Agentic Interaction Framework for Skeleton-based Action Recognition](https://mlsanity.com/paper/2511.22433/) (2025)
- [Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challenges](https://mlsanity.com/paper/2510.19292/) (2025)

## Abstract (arXiv)

This study investigates unsupervised anomaly action recognition, which identifies video-level abnormal-human-behavior events in an unsupervised manner without abnormal samples, and simultaneously addresses three limitations in the conventional skeleton-based approaches: target domain-dependent DNN training, robustness against skeleton errors, and a lack of normal samples. We present a unified, user prompt-guided zero-shot learning framework using a target domain-independent skeleton feature extractor, which is pretrained on a large-scale action recognition dataset. Particularly, during the training phase using normal samples, the method models the distribution of skeleton features of the normal actions while freezing the weights of the DNNs and estimates the anomaly score using this distribution in the inference phase. Additionally, to increase robustness against skeleton errors, we introduce a DNN architecture inspired by a point cloud deep learning paradigm, which sparsely propagates the features between joints. Furthermore, to prevent the unobserved normal actions from being misidentified as abnormal actions, we incorporate a similarity score between the user prompt embeddings and skeleton features aligned in the common space into the anomaly score, which indirectly supplements normal actions. On two publicly available datasets, we conduct experiments to test the effectiveness of the proposed method with respect to abovementioned limitations.