# Context-Aware RCNN: A Baseline for Action Detection in Videos

arXiv: https://arxiv.org/abs/2007.09861 · HTML: https://mlsanity.com/paper/2007.09861/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2020.

## Built on by (most cited first, 30 of 33)

- [MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions](https://mlsanity.com/paper/2105.07404/) (2021)
- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [Holistic Interaction Transformer Network for Action Detection](https://mlsanity.com/paper/2210.12686/) (2022)
- [STMixer: A One-Stage Sparse Action Detector](https://mlsanity.com/paper/2303.15879/) (2023)
- [Efficient Video Action Detection with Token Dropout and Context Refinement](https://mlsanity.com/paper/2304.08451/) (2023)
- [ACGNet: Action Complement Graph Network for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2112.10977/) (2021)
- [SportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videos](https://mlsanity.com/paper/2404.04565/) (2024)
- [Panoramic Human Activity Recognition](https://mlsanity.com/paper/2203.03806/) (2022)
- [Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection](https://mlsanity.com/paper/2411.10922/) (2024)
- [A Simple and Efficient Pipeline to Build an End-to-End Spatial-Temporal Action Detector](https://mlsanity.com/paper/2206.03064/) (2022)
- [YOWOv2: A Stronger yet Efficient Multi-level Detection Framework for Real-time Spatio-temporal Action Detection](https://mlsanity.com/paper/2302.06848/) (2023)
- [A Survey on Deep Learning-based Spatio-temporal Action Detection](https://mlsanity.com/paper/2308.01618/) (2023)
- [Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies](https://mlsanity.com/paper/2410.08860/) (2024)
- [Multiscale Vision Transformers meet Bipartite Matching for efficient single-stage Action Localization](https://mlsanity.com/paper/2312.17686/) (2023)
- [Identity-aware Graph Memory Network for Action Detection](https://mlsanity.com/paper/2108.11559/) (2021)
- [Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/2206.10095/) (2022)
- [Explore Spatio-temporal Aggregation for Insubstantial Object Detection: Benchmark Dataset and Baseline](https://mlsanity.com/paper/2206.11459/) (2022)
- [Time and Frequency Network for Human Action Detection in Videos](https://mlsanity.com/paper/2103.04680/) (2021)
- [Spatio-Temporal Context for Action Detection](https://mlsanity.com/paper/2106.15171/) (2021)
- [LSTC: Boosting Atomic Action Detection with Long-Short-Term Context](https://mlsanity.com/paper/2110.09819/) (2021)
- [Point3D: tracking actions as moving points with 3D CNNs](https://mlsanity.com/paper/2203.10584/) (2022)
- [MM-SEAL: A Large-scale Video Dataset of Multi-person Multi-grained Spatio-temporally Action Localization](https://mlsanity.com/paper/2204.02688/) (2022)
- [E^2TAD: An Energy-Efficient Tracking-based Action Detector](https://mlsanity.com/paper/2204.04416/) (2022)
- [E^2VTS: Energy-Efficient Video Text Spotting from Unmanned Aerial Vehicles](https://mlsanity.com/paper/2206.02281/) (2022)
- [DOAD: Decoupled One Stage Action Detection Network](https://mlsanity.com/paper/2304.00254/) (2023)
- [Unified Framework with Consistency across Modalities for Human Activity Recognition](https://mlsanity.com/paper/2409.02385/) (2024)
- [Gaseous Object Detection](https://mlsanity.com/paper/2502.12415/) (2025)
- [Dual-Path Temporal Map Optimization for Make-up Temporal Video Grounding](https://mlsanity.com/paper/2309.06176/) (2023)
- [A Grammatical Compositional Model for Video Action Detection](https://mlsanity.com/paper/2310.02887/) (2023)

## Abstract (arXiv)

Video action detection approaches usually conduct actor-centric action recognition over RoI-pooled features following the standard pipeline of Faster-RCNN. In this work, we first empirically find the recognition accuracy is highly correlated with the bounding box size of an actor, and thus higher resolution of actors contributes to better performance. However, video models require dense sampling in time to achieve accurate recognition. To fit in GPU memory, the frames to backbone network must be kept low-resolution, resulting in a coarse feature map in RoI-Pooling layer. Thus, we revisit RCNN for actor-centric action recognition via cropping and resizing image patches around actors before feature extraction with I3D deep network. Moreover, we found that expanding actor bounding boxes slightly and fusing the context features can further boost the performance. Consequently, we develop a surpringly effective baseline (Context-Aware RCNN) and it achieves new state-of-the-art results on two challenging action detection benchmarks of AVA and JHMDB. Our observations challenge the conventional wisdom of RoI-Pooling based pipeline and encourage researchers rethink the importance of resolution in actor-centric action recognition. Our approach can serve as a strong baseline for video action detection and is expected to inspire new ideas for this filed. The code is available at urlhttps://github.com/MCG-NJU/CRCNN-Action.