# PO-GUISE+: Pose and object guided transformer token selection for efficient driver action recognition

arXiv: https://arxiv.org/abs/2407.13750 · HTML: https://mlsanity.com/paper/2407.13750/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 69 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 30 of 31)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [Spatiotemporal Residual Networks for Video Action Recognition](https://mlsanity.com/paper/1611.02155/) (2016)
- [Which Tokens to Use? Investigating Token Reduction in Vision Transformers](https://mlsanity.com/paper/2308.04657/) (2023)
- [DMD: A Large-Scale Multi-Modal Driver Monitoring Dataset for Attention and Alertness Analysis](https://mlsanity.com/paper/2008.12085/) (2020)
- [STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition](https://mlsanity.com/paper/2210.07503/) (2022)
- [PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimation](https://mlsanity.com/paper/2209.08194/) (2022)
- [TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration](https://mlsanity.com/paper/2203.00927/) (2022)
- [Cross-Modal Learning with 3D Deformable Attention for Action Recognition](https://mlsanity.com/paper/2212.05638/) (2022)
- [Efficient Video Action Detection with Token Dropout and Context Refinement](https://mlsanity.com/paper/2304.08451/) (2023)
- [Just Add π! Pose Induced Video Transformers for Understanding Activities of Daily Living](https://mlsanity.com/paper/2311.18840/) (2023)
- [Vision-Language Models can Identify Distracted Driver Behavior from Naturalistic Videos](https://mlsanity.com/paper/2306.10159/) (2023)
- [Pose And Joint-Aware Action Recognition](https://mlsanity.com/paper/2010.08164/) (2020)
- [Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition](https://mlsanity.com/paper/2101.06636/) (2021)
- [Video Action Recognition with Attentive Semantic Units](https://mlsanity.com/paper/2303.09756/) (2023)
- [Motion-driven Visual Tempo Learning for Video-based Action Recognition](https://mlsanity.com/paper/2202.12116/) (2022)

## Built on by (most cited first, 2 of 2)

- [DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting](https://mlsanity.com/paper/2609.08117/) (2026)
- [MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents](https://mlsanity.com/paper/2606.31167/) (2026)

## Abstract (arXiv)

We address the task of identifying distracted driving by analyzing in-car videos using efficient transformers. Although transformer models have achieved outstanding performance in human action recognition tasks, their high computational costs limit their application onboard a vehicle. We introduce POGUISE+, a multi-task video transformer that, given an input clip, predicts the distracted driving action, the driver's pose, and the interacting object. Our enhanced features for token selection are specifically adapted to driver actions by leveraging information about object interaction and the driver's pose. With POGUISE+, we significantly reduce the model's computational demands while maintaining or improving baseline accuracy across various computational budgets. Additionally, to evaluate our model's performance in real-world scenarios, we have developed benchmarks on a Jetson computing platform, demonstrating its effectiveness across different configurations and computational budgets. Our model outperforms current state-of-the-art results on the Drive&Act, 100-Driver, and 3MDAD datasets, while having superior efficiency compared to existing video transformer-based methods.