Human-centric Behavior Description in Videos: New Benchmark and Model
Not on the map. Why?
Abstract (arXiv)
In the domain of video surveillance, describing the behavior of each individual within the video is becoming increasingly essential, especially in complex scenarios with multiple individuals present. This is because describing each individual's behavior provides more detailed situational analysis, enabling accurate assessment and response to potential risks, ensuring the safety and harmony of public places. Currently, video-level captioning datasets cannot provide fine-grained descriptions for each individual's specific behavior. However, mere descriptions at the video-level fail to provide an in-depth interpretation of individual behaviors, making it challenging to accurately determine the specific identity of each individual. To address this challenge, we construct a human-centric video surveillance captioning dataset, which provides detailed descriptions of the dynamic behaviors of 7,820 individuals. Specifically, we have labeled several aspects of each person, such as location, clothing, and interactions with other elements in the scene, and these people are distributed across 1,012 videos. Based on this dataset, we can link individuals to their respective behaviors, allowing for further analysis of each person's behavior in surveillance videos. Besides the dataset, we propose a novel video captioning approach that can describe individual behavior in detail on a person-level basis, achieving state-of-the-art results. To facilitate further research in this field, we intend to release our dataset and code.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 46 of its 86 references that are in the map. First posted on arXiv 4 Oct 2023.
Builds on
46 of 86 references.
References · 46
- 2017Attention Is All You Needcited by 59,932
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
- 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
- 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
- 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
- 2017Dense-Captioning Events in Videoscited by 789
34 more
- 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
- 2018Real-world Anomaly Detection in Surveillance Videoscited by 570
- 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
- 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
- 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
- 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
- 2016Learning Temporal Regularity in Video Sequencescited by 254
- 2020Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervisioncited by 235
- 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
- 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
- 2022StrongSORT: Make DeepSORT Great Againcited by 213
- 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
- 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
- 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
- 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
- 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
- 2019Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videoscited by 103
- 2019What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessmentcited by 100
- 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
- 2018Anomaly Detection using One-Class Neural Networkscited by 82
- 2020A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformercited by 75
- 2018Bidirectional Attentive Fusion with Context Gating for Dense Video Captioningcited by 68
- 2018Jointly Localizing and Describing Events for Dense Video Captioningcited by 60
- 2023Video Event Restoration Based on Keyframes for Video Anomaly Detectioncited by 54
- 2016Deep-Anomaly: Fully Convolutional Neural Network for Fast Anomaly Detection in Crowded Scenescited by 50
- 2017Unmasking the abnormal events in videocited by 48
- 2023Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challengescited by 42
- 2018Video Anomaly Detection and Localization via Gaussian Mixture Fully Convolutional Variational Autoencodercited by 21
- 2015Real-Time Anomaly Detection and Localization in Crowded Scenescited by 20
- 2022GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrievalcited by 19
- 2023VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blendingcited by 17
- 2022Self-Training Vision Language BERTs with a Unified Conditional Modelcited by 6
- 2022Diverse Video Captioning by Adaptive Spatio-temporal Attentioncited by 2
- 2015Modeling Representation of Videos for Anomaly Detection using Deep Learning: A Reviewcited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.