# Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition

arXiv: https://arxiv.org/abs/2101.06636 · HTML: https://mlsanity.com/paper/2101.06636/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 61 references that are in the map.
- First posted on arXiv 17 Jan 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning](https://mlsanity.com/paper/1811.02307/) (2018)
- [Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos](https://mlsanity.com/paper/1507.05738/) (2015)
- [Timeception for Complex Action Recognition](https://mlsanity.com/paper/1812.01289/) (2018)
- [Long-term Temporal Convolutions for Action Recognition](https://mlsanity.com/paper/1604.04494/) (2016)
- [Temporal Recurrent Networks for Online Action Detection](https://mlsanity.com/paper/1811.07391/) (2018)
- [ActionVLAD: Learning spatio-temporal aggregation for action classification](https://mlsanity.com/paper/1704.02895/) (2017)
- [Memory-Attended Recurrent Network for Video Captioning](https://mlsanity.com/paper/1905.03966/) (2019)
- [Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks](https://mlsanity.com/paper/1603.07772/) (2016)
- [Learnable pooling with Context Gating for video classification](https://mlsanity.com/paper/1706.06905/) (2017)
- [VideoGraph: Recognizing Minutes-Long Human Activities in Videos](https://mlsanity.com/paper/1905.05143/) (2019)
- [Attentional Pooling for Action Recognition](https://mlsanity.com/paper/1711.01467/) (2017)
- [Action Recognition using Visual Attention](https://mlsanity.com/paper/1511.04119/) (2015)
- [Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks](https://mlsanity.com/paper/1704.02581/) (2017)

## Built on by (most cited first, 14 of 14)

- [TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration](https://mlsanity.com/paper/2203.00927/) (2022)
- [A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding](https://mlsanity.com/paper/2204.04734/) (2022)
- [Multi-modality action recognition based on dual feature shift in vehicle cabin monitoring](https://mlsanity.com/paper/2401.14838/) (2024)
- [CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognition](https://mlsanity.com/paper/2406.11340/) (2024)
- [Stacked Temporal Attention: Improving First-person Action Recognition by Emphasizing Discriminative Clips](https://mlsanity.com/paper/2112.01038/) (2021)
- [Is my Driver Observation Model Overconfident? Input-guided Calibration Networks for Reliable and Interpretable Confidence Estimates](https://mlsanity.com/paper/2204.04674/) (2022)
- [MultiFuser: Multimodal Fusion Transformer for Enhanced Driver Action Recognition](https://mlsanity.com/paper/2408.01766/) (2024)
- [PO-GUISE+: Pose and object guided transformer token selection for efficient driver action recognition](https://mlsanity.com/paper/2407.13750/) (2024)
- [Probing Fine-Grained Action Understanding and Cross-View Generalization of Foundation Models](https://mlsanity.com/paper/2407.15605/) (2024)
- [Gesture Recognition with Keypoint and Radar Stream Fusion for Automated Vehicles](https://mlsanity.com/paper/2302.09998/) (2023)
- [DSDFormer: An Innovative Transformer-Mamba Framework for Robust High-Precision Driver Distraction Identification](https://mlsanity.com/paper/2409.05587/) (2024)
- [Mixture-of-Modality-Experts with Holistic Token Learning for Fine-Grained Multimodal Visual Analytics in Driver Action Recognition](https://mlsanity.com/paper/2604.05947/) (2026)
- [MIFI: MultI-camera Feature Integration for Roust 3D Distracted Driver Activity Recognition](https://mlsanity.com/paper/2401.14115/) (2024)
- [T-MASK: Temporal Masking for Probing Foundation Models across Camera Views in Driver Monitoring](https://mlsanity.com/paper/2508.16207/) (2025)

## Abstract (arXiv)

There is significant progress in recognizing traditional human activities from videos focusing on highly distinctive actions involving discriminative body movements, body-object and/or human-human interactions. Driver's activities are different since they are executed by the same subject with similar body parts movements, resulting in subtle changes. To address this, we propose a novel framework by exploiting the spatiotemporal attention to model the subtle changes. Our model is named Coarse Temporal Attention Network (CTA-Net), in which coarse temporal branches are introduced in a trainable glimpse network. The goal is to allow the glimpse to capture high-level temporal relationships, such as 'during', 'before' and 'after' by focusing on a specific part of a video. These branches also respect the topology of the temporal dynamics in the video, ensuring that different branches learn meaningful spatial and temporal changes. The model then uses an innovative attention mechanism to generate high-level action specific contextual information for activity recognition by exploring the hidden states of an LSTM. The attention mechanism helps in learning to decide the importance of each hidden state for the recognition task by weighing them when constructing the representation of the video. Our approach is evaluated on four publicly accessible datasets and significantly outperforms the state-of-the-art by a considerable margin with only RGB video as input.