# Action Recognition using Visual Attention

arXiv: https://arxiv.org/abs/1511.04119 · HTML: https://mlsanity.com/paper/1511.04119/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2015.

## Built on by (most cited first, 30 of 49)

- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [Temporal Context Aggregation Network for Temporal Action Proposal Refinement](https://mlsanity.com/paper/2103.13141/) (2021)
- [Differentiable Patch Selection for Image Recognition](https://mlsanity.com/paper/2104.03059/) (2021)
- [MaskTune: Mitigating Spurious Correlations by Forcing to Explore](https://mlsanity.com/paper/2210.00055/) (2022)
- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Class Semantics-based Attention for Action Detection](https://mlsanity.com/paper/2109.02613/) (2021)
- [Attention Mechanism in Neural Networks: Where it Comes and Where it Goes](https://mlsanity.com/paper/2204.13154/) (2022)
- [GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction](https://mlsanity.com/paper/2110.00380/) (2021)
- [TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Device](https://mlsanity.com/paper/2109.13227/) (2021)
- [Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition](https://mlsanity.com/paper/2101.06636/) (2021)
- [Understanding the computational demands underlying visual reasoning](https://mlsanity.com/paper/2108.03603/) (2021)
- [Binding Actions to Objects in World Models](https://mlsanity.com/paper/2204.13022/) (2022)
- [A Hierarchical Conditional Random Field-based Attention Mechanism Approach for Gastric Histopathology Image Classification](https://mlsanity.com/paper/2102.10499/) (2021)
- [Towards Extremely Compact RNNs for Video Recognition with Fully Decomposed Hierarchical Tucker Structure](https://mlsanity.com/paper/2104.05758/) (2021)
- [Attention mechanisms and deep learning for machine vision: A survey of the state of the art](https://mlsanity.com/paper/2106.07550/) (2021)
- [Towards Improved Input Masking for Convolutional Neural Networks](https://mlsanity.com/paper/2211.14646/) (2022)
- [Video-based Human Action Recognition using Deep Learning: A Review](https://mlsanity.com/paper/2208.03775/) (2022)
- [Learning to Recognize Actions on Objects in Egocentric Video with Attention Dictionaries](https://mlsanity.com/paper/2102.08065/) (2021)
- [Trends in Vehicle Re-identification Past, Present, and Future: A Comprehensive Review](https://mlsanity.com/paper/2102.09744/) (2021)
- [Multimodal Continuous Visual Attention Mechanisms](https://mlsanity.com/paper/2104.03046/) (2021)
- [Information Bottleneck Approach to Spatial Attention Learning](https://mlsanity.com/paper/2108.03418/) (2021)
- [Video Violence Recognition and Localization Using a Semi-Supervised Hard Attention Model](https://mlsanity.com/paper/2202.02212/) (2022)
- [Planning on the fast lane: Learning to interact using attention mechanisms in path integral inverse reinforcement learning](https://mlsanity.com/paper/2007.05798/) (2020)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [On the Post-hoc Explainability of Deep Echo State Networks for Time Series Forecasting, Image and Video Classification](https://mlsanity.com/paper/2102.08634/) (2021)
- [Actor-centered Representations for Action Localization in Streaming Videos](https://mlsanity.com/paper/2104.14131/) (2021)
- [VideoLightFormer: Lightweight Action Recognition using Transformers](https://mlsanity.com/paper/2107.00451/) (2021)
- [Deep Reinforcement Learning for Data-Driven Adaptive Scanning in Ptychography](https://mlsanity.com/paper/2203.15413/) (2022)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)

## Abstract (arXiv)

We propose a soft attention based model for the task of action recognition in videos. We use multi-layered Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units which are deep both spatially and temporally. Our model learns to focus selectively on parts of the video frames and classifies videos after taking a few glimpses. The model essentially learns which parts in the frames are relevant for the task at hand and attaches higher importance to them. We evaluate the model on UCF-11 (YouTube Action), HMDB-51 and Hollywood2 datasets and analyze how the model focuses its attention depending on the scene and the action being performed.