# Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification

arXiv: https://arxiv.org/abs/1808.01106 · HTML: https://mlsanity.com/paper/1808.01106/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2018.

## Built on by (most cited first, 6 of 6)

- [The State of Aerial Surveillance: A Survey](https://mlsanity.com/paper/2201.03080/) (2022)
- [Panoramic Human Activity Recognition](https://mlsanity.com/paper/2203.03806/) (2022)
- [Temporal Transformer Networks with Self-Supervision for Action Recognition](https://mlsanity.com/paper/2112.07338/) (2021)
- [A Multi-Modal Transformer Network for Action Detection](https://mlsanity.com/paper/2305.19624/) (2023)
- [3D Convolutional Networks for Action Recognition: Application to Sport Gesture Recognition](https://mlsanity.com/paper/2204.08460/) (2022)
- [A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection](https://mlsanity.com/paper/2405.08204/) (2024)

## Abstract (arXiv)

Local features at neighboring spatial positions in feature maps have high correlation since their receptive fields are often overlapped. Self-attention usually uses the weighted sum (or other functions) with internal elements of each local feature to obtain its weight score, which ignores interactions among local features. To address this, we propose an effective interaction-aware self-attention model inspired by PCA to learn attention maps. Furthermore, since different layers in a deep network capture feature maps of different scales, we use these feature maps to construct a spatial pyramid and then utilize multi-scale information to obtain more accurate attention scores, which are used to weight the local features in all spatial positions of feature maps to calculate attention maps. Moreover, our spatial pyramid attention is unrestricted to the number of its input feature maps so it is easily extended to a spatio-temporal version. Finally, our model is embedded in general CNNs to form end-to-end attention networks for action classification. Experimental results show that our method achieves the state-of-the-art results on the UCF101, HMDB51 and untrimmed Charades.