# Beyond Walking: A Large-Scale Image-Text Benchmark for Text-based Person Anomaly Search

arXiv: https://arxiv.org/abs/2411.17776 · HTML: https://mlsanity.com/paper/2411.17776/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 75 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in vitro](https://mlsanity.com/paper/1701.07717/) (2017)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [Joint Discriminative and Generative Learning for Person Re-identification](https://mlsanity.com/paper/1904.07223/) (2019)
- [Person Search with Natural Language Description](https://mlsanity.com/paper/1702.05729/) (2017)
- [Dual-Path Convolutional Image-Text Embeddings with Instance Loss](https://mlsanity.com/paper/1711.05535/) (2017)
- [Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval](https://mlsanity.com/paper/2303.12501/) (2023)
- [Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification](https://mlsanity.com/paper/2107.12666/) (2021)
- [DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval](https://mlsanity.com/paper/2109.05534/) (2021)
- [Generative Cooperative Learning for Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2203.03962/) (2022)
- [Improving Person Re-identification by Attribute and Identity Learning](https://mlsanity.com/paper/1703.07220/) (2017)
- [Street Scene: A new dataset and evaluation protocol for video anomaly detection](https://mlsanity.com/paper/1902.05872/) (2019)
- [Oops! Predicting Unintentional Action in Video](https://mlsanity.com/paper/1911.11206/) (2019)
- [CLIP-Driven Fine-grained Text-Image Person Re-identification](https://mlsanity.com/paper/2210.10276/) (2022)
- [Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark](https://mlsanity.com/paper/2306.02898/) (2023)
- [See Finer, See More: Implicit Modality Alignment for Text-based Person Retrieval](https://mlsanity.com/paper/2208.08608/) (2022)
- [Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments](https://mlsanity.com/paper/1906.09610/) (2019)
- [Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search](https://mlsanity.com/paper/2101.03036/) (2021)
- [Learning Granularity-Unified Representations for Text-to-Image Person Re-identification](https://mlsanity.com/paper/2207.07802/) (2022)
- [Normalizing Flows for Human Pose Anomaly Detection](https://mlsanity.com/paper/2211.10946/) (2022)
- [TIPCB: A Simple but Effective Part-based Convolutional Baseline for Text-based Person Search](https://mlsanity.com/paper/2105.11628/) (2021)

## Built on by (most cited first, 11 of 11)

- [Hybrid, Unified and Iterative: A Novel Framework for Text-based Person Anomaly Retrieval](https://mlsanity.com/paper/2511.22470/) (2025)
- [AnomalyLMM: Bridging Generative Knowledge and Discriminative Retrieval for Text-Based Person Anomaly Search](https://mlsanity.com/paper/2509.04376/) (2025)
- [Bridging the Pose-Semantic Gap: A Cascade Framework for Text-Based Person Anomaly Search](https://mlsanity.com/paper/2604.23282/) (2026)
- [Minimizing the Pretraining Gap: Domain-aligned Text-Based Person Retrieval](https://mlsanity.com/paper/2507.10195/) (2025)
- [Efficient Vision Language Model Fine-tuning for Text-based Person Anomaly Search](https://mlsanity.com/paper/2502.03230/) (2025)
- [FaLCon: Facet-Anchored Retrieval with Late Consensus for Sim2Real Text-Based Person Anomaly Search](https://mlsanity.com/paper/2608.09474/) (2026)
- [Heterogeneous Vision-Language Ensemble with Disagreement-Aware Reranking for Text-Based Person Anomaly Retrieval](https://mlsanity.com/paper/2608.12843/) (2026)
- [Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search](https://mlsanity.com/paper/2604.08598/) (2026)
- [Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Search](https://mlsanity.com/paper/2608.23503/) (2026)
- [SCOUT: Sim-to-Real Text-Based Person Retrieval by Embedding-Space Prediction over Frozen Video Features](https://mlsanity.com/paper/2609.19483/) (2026)
- [Absence is Presence: Understanding Visual Scene Negative Events Under Safety Cognitive Constraint](https://mlsanity.com/paper/2609.19812/) (2026)

## Abstract (arXiv)

Text-based person search aims to retrieve specific individuals across camera networks using natural language descriptions. However, current benchmarks often exhibit biases towards common actions like walking or standing, neglecting the critical need for identifying abnormal behaviors in real-world scenarios. To meet such demands, we propose a new task, text-based person anomaly search, locating pedestrians engaged in both routine or anomalous activities via text. To enable the training and evaluation of this new task, we construct a large-scale image-text Pedestrian Anomaly Behavior (PAB) benchmark, featuring a broad spectrum of actions, e.g., running, performing, playing soccer, and the corresponding anomalies, e.g., lying, being hit, and falling of the same identity. The training set of PAB comprises 1,013,605 synthesized image-text pairs of both normalities and anomalies, while the test set includes 1,978 real-world image-text pairs. To validate the potential of PAB, we introduce a cross-modal pose-aware framework, which integrates human pose patterns with identity-based hard negative pair sampling. Extensive experiments on the proposed benchmark show that synthetic training data facilitates the fine-grained behavior retrieval, and the proposed pose-aware method arrives at 84.93% recall@1 accuracy, surpassing other competitive methods. The dataset, model, and code are available at https://github.com/Shuyu-XJTU/CMP.