# Contextual Action Recognition with R*CNN

arXiv: https://arxiv.org/abs/1505.01197 · HTML: https://mlsanity.com/paper/1505.01197/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2015.

## Built on by (most cited first, 30 of 34)

- [Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos](https://mlsanity.com/paper/2507.15597/) (2025)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Unified Graph Structured Models for Video Understanding](https://mlsanity.com/paper/2103.15662/) (2021)
- [The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods](https://mlsanity.com/paper/2104.03178/) (2021)
- [Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion](https://mlsanity.com/paper/2312.10692/) (2023)
- [From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding](https://mlsanity.com/paper/2304.00553/) (2023)
- [Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning](https://mlsanity.com/paper/2311.17365/) (2023)
- [An Empirical Study of Mamba-based Pedestrian Attribute Recognition](https://mlsanity.com/paper/2407.10374/) (2024)
- [Are all outliers alike? On Understanding the Diversity of Outliers for Detecting OODs](https://mlsanity.com/paper/2103.12628/) (2021)
- [Attend and Guide (AG-Net): A Keypoints-driven Attention-based Deep Network for Image Recognition](https://mlsanity.com/paper/2110.12183/) (2021)
- [The Overlooked Classifier in Human-Object Interaction Recognition](https://mlsanity.com/paper/2112.06392/) (2021)
- [Detecting OODs as datapoints with High Uncertainty](https://mlsanity.com/paper/2108.06380/) (2021)
- [Using Semantic Information for Defining and Detecting OOD Inputs](https://mlsanity.com/paper/2302.11019/) (2023)
- [Multi-Expert Human Action Recognition with Hierarchical Super-Class Learning](https://mlsanity.com/paper/2112.07015/) (2021)
- [RNNs, CNNs and Transformers in Human Action Recognition: A Survey and a Hybrid Model](https://mlsanity.com/paper/2407.06162/) (2024)
- [Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?](https://mlsanity.com/paper/2512.02846/) (2025)
- [Regional Attention Network (RAN) for Head Pose and Fine-grained Gesture Recognition](https://mlsanity.com/paper/2101.06634/) (2021)
- [Modeling long-term interactions to enhance action recognition](https://mlsanity.com/paper/2104.11520/) (2021)
- [Human Object Interaction Detection using Two-Direction Spatial Enhancement and Exclusive Object Prior](https://mlsanity.com/paper/2105.03089/) (2021)
- [Is Object Detection Necessary for Human-Object Interaction Recognition?](https://mlsanity.com/paper/2107.13083/) (2021)
- [Human Action Recognition in Still Images Using ConViT](https://mlsanity.com/paper/2307.08994/) (2023)
- [DP-Net: Learning Discriminative Parts for image recognition](https://mlsanity.com/paper/2404.15037/) (2024)
- [A Survey on Backbones for Deep Video Action Recognition](https://mlsanity.com/paper/2405.05584/) (2024)
- [Understanding Multimodal Complementarity for Single-Frame Action Anticipation](https://mlsanity.com/paper/2601.22039/) (2026)
- [Bitstream Action Recognition is Byte Modeling](https://mlsanity.com/paper/2608.15695/) (2026)
- [LR-Net: A Block-based Convolutional Neural Network for Low-Resolution Image Classification](https://mlsanity.com/paper/2207.09531/) (2022)
- [Autonomous Navigation in Dynamic Environments: Deep Learning-Based Approach](https://mlsanity.com/paper/2102.08758/) (2021)
- [Fight Detection from Still Images in the Wild](https://mlsanity.com/paper/2111.08370/) (2021)
- [The Overlooked Classifier in Human-Object Interaction Recognition](https://mlsanity.com/paper/2203.05676/) (2022)
- [Detection of Degraded Acacia tree species using deep neural networks on uav drone imagery](https://mlsanity.com/paper/2204.07096/) (2022)

## Abstract (arXiv)

There are multiple cues in an image which reveal what action a person is performing. For example, a jogger has a pose that is characteristic for jogging, but the scene (e.g. road, trail) and the presence of other joggers can be an additional source of information. In this work, we exploit the simple observation that actions are accompanied by contextual cues to build a strong action recognition system. We adapt RCNN to use more than one region for classification while still maintaining the ability to localize the action. We call our system R*CNN. The action-specific models and the feature maps are trained jointly, allowing for action specific representations to emerge. R*CNN achieves 90.2% mean AP on the PASAL VOC Action dataset, outperforming all other approaches in the field by a significant margin. Last, we show that R*CNN is not limited to action recognition. In particular, R*CNN can also be used to tackle fine-grained tasks such as attribute classification. We validate this claim by reporting state-of-the-art performance on the Berkeley Attributes of People dataset.