# Human-Centric Video Anomaly Detection Through Spatio-Temporal Pose Tokenization and Transformer

arXiv: https://arxiv.org/abs/2408.15185 · HTML: https://mlsanity.com/paper/2408.15185/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 101 references that are in the map.
- First posted on arXiv 27 Aug 2024.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [Anomaly Detection in Video via Self-Supervised and Multi-Task Learning](https://mlsanity.com/paper/2011.07491/) (2020)
- [Representational Strengths and Limitations of Transformers](https://mlsanity.com/paper/2306.02896/) (2023)
- [Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detection](https://mlsanity.com/paper/2111.09099/) (2021)
- [A Survey of Visual Transformers](https://mlsanity.com/paper/2111.06091/) (2021)
- [Old is Gold: Redefining the Adversarially Learned One-Class Classifier Training Paradigm](https://mlsanity.com/paper/2004.07657/) (2020)
- [Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases](https://mlsanity.com/paper/2010.15052/) (2020)
- [Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos](https://mlsanity.com/paper/1903.03295/) (2019)
- [Generative Cooperative Learning for Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2203.03962/) (2022)
- [Graph Embedded Pose Clustering for Anomaly Detection](https://mlsanity.com/paper/1912.11850/) (2019)
- [Lite-HRNet: A Lightweight High-Resolution Network](https://mlsanity.com/paper/2104.06403/) (2021)
- [MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2211.15098/) (2022)
- [A Survey of Single-Scene Video Anomaly Detection](https://mlsanity.com/paper/2004.05993/) (2020)
- [A Background-Agnostic Framework with Adversarial Training for Abnormal Event Detection in Video](https://mlsanity.com/paper/2008.12328/) (2020)
- [Learning Normal Dynamics in Videos with Meta Prototype Network](https://mlsanity.com/paper/2104.06689/) (2021)
- [Video Anomaly Detection by Solving Decoupled Spatio-Temporal Jigsaw Puzzles](https://mlsanity.com/paper/2207.10172/) (2022)
- [Normalizing Flows for Human Pose Anomaly Detection](https://mlsanity.com/paper/2211.10946/) (2022)
- [Multi-timescale Trajectory Prediction for Abnormal Human Activity Detection](https://mlsanity.com/paper/1908.04321/) (2019)
- [SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection](https://mlsanity.com/paper/2207.08003/) (2022)
- [A New Comprehensive Benchmark for Semi-supervised Video Anomaly Detection and Anticipation](https://mlsanity.com/paper/2305.13611/) (2023)
- [Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks](https://mlsanity.com/paper/1710.09207/) (2017)
- [Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection](https://mlsanity.com/paper/2209.12148/) (2022)
- [A Hierarchical Spatio-Temporal Graph Convolutional Neural Network for Anomaly Detection in Videos](https://mlsanity.com/paper/2112.04294/) (2021)

## Built on by (most cited first, 6 of 6)

- [Shopformer: Transformer-Based Framework for Detecting Shoplifting via Human Pose](https://mlsanity.com/paper/2504.19970/) (2025)
- [A Survey on Video Anomaly Detection via Deep Learning: Human, Vehicle, and Environment](https://mlsanity.com/paper/2508.14203/) (2025)
- [ALFred: An Active Learning Framework for Real-world Semi-supervised Anomaly Detection with Adaptive Thresholds](https://mlsanity.com/paper/2508.09058/) (2025)
- [From Offline to Periodic Adaptation for Pose-Based Shoplifting Detection in Real-world Retail Security](https://mlsanity.com/paper/2603.04723/) (2026)
- [Are Multimodal LLMs Ready for Surveillance? A Reality Check on Zero-Shot Anomaly Detection in the Wild](https://mlsanity.com/paper/2603.04727/) (2026)
- [From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detection](https://mlsanity.com/paper/2604.09327/) (2026)

## Abstract (arXiv)

Video Anomaly Detection (VAD) presents a significant challenge in computer vision, particularly due to the unpredictable and infrequent nature of anomalous events, coupled with the diverse and dynamic environments in which they occur. Human-centric VAD, a specialized area within this domain, faces additional complexities, including variations in human behavior, potential biases in data, and substantial privacy concerns related to human subjects. These issues complicate the development of models that are both robust and generalizable. To address these challenges, recent advancements have focused on pose-based VAD, which leverages human pose as a high-level feature to mitigate privacy concerns, reduce appearance biases, and minimize background interference. In this paper, we introduce SPARTA, a novel transformer-based architecture designed specifically for human-centric pose-based VAD. SPARTA introduces an innovative Spatio-Temporal Pose and Relative Pose (ST-PRP) tokenization method that produces an enriched representation of human motion over time. This approach ensures that the transformer's attention mechanism captures both spatial and temporal patterns simultaneously, rather than focusing on only one aspect. The addition of the relative pose further emphasizes subtle deviations from normal human movements. The architecture's core, a novel Unified Encoder Twin Decoders (UETD) transformer, significantly improves the detection of anomalous behaviors in video data. Extensive evaluations across multiple benchmark datasets demonstrate that SPARTA consistently outperforms existing methods, establishing a new state-of-the-art in pose-based VAD.