# Multi-modality action recognition based on dual feature shift in vehicle cabin monitoring

arXiv: https://arxiv.org/abs/2401.14838 · HTML: https://mlsanity.com/paper/2401.14838/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 26 Jan 2024.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition](https://mlsanity.com/paper/2101.06636/) (2021)
- [A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding](https://mlsanity.com/paper/2204.04734/) (2022)
- [Supervised Contrastive Learning for Detecting Anomalous Driving Behaviours from Multimodal Videos](https://mlsanity.com/paper/2109.04021/) (2021)
- [Real-Time Driver Monitoring Systems through Modality and View Analysis](https://mlsanity.com/paper/2210.09441/) (2022)
- [SkateboardAI: The Coolest Video Action Recognition for Skateboarding](https://mlsanity.com/paper/2311.11467/) (2023)
- [Artificial Intelligence Methods in In-Cabin Use Cases: A Survey](https://mlsanity.com/paper/2101.02082/) (2021)

## Built on by (most cited first, 8 of 8)

- [CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognition](https://mlsanity.com/paper/2406.11340/) (2024)
- [MultiFuser: Multimodal Fusion Transformer for Enhanced Driver Action Recognition](https://mlsanity.com/paper/2408.01766/) (2024)
- [A Comprehensive Methodological Survey of Human Activity Recognition Across Divers Data Modalities](https://mlsanity.com/paper/2409.09678/) (2024)
- [DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Dataset](https://mlsanity.com/paper/2601.11990/) (2026)
- [Mixture-of-Modality-Experts with Holistic Token Learning for Fine-Grained Multimodal Visual Analytics in Driver Action Recognition](https://mlsanity.com/paper/2604.05947/) (2026)
- [Spatial-Temporal Perception with Causal Inference for Naturalistic Driving Action Recognition](https://mlsanity.com/paper/2503.04078/) (2025)
- [T-MASK: Temporal Masking for Probing Foundation Models across Camera Views in Driver Monitoring](https://mlsanity.com/paper/2508.16207/) (2025)
- [Multi-View In-Cabin Monitoring System for Public Transport Vehicles](https://mlsanity.com/paper/2606.11739/) (2026)

## Abstract (arXiv)

Driver Action Recognition (DAR) is crucial in vehicle cabin monitoring systems. In real-world applications, it is common for vehicle cabins to be equipped with cameras featuring different modalities. However, multi-modality fusion strategies for the DAR task within car cabins have rarely been studied. In this paper, we propose a novel yet efficient multi-modality driver action recognition method based on dual feature shift, named DFS. DFS first integrates complementary features across modalities by performing modality feature interaction. Meanwhile, DFS achieves the neighbour feature propagation within single modalities, by feature shifting among temporal frames. To learn common patterns and improve model efficiency, DFS shares feature extracting stages among multiple modalities. Extensive experiments have been carried out to verify the effectiveness of the proposed DFS model on the Drive&Act dataset. The results demonstrate that DFS achieves good performance and improves the efficiency of multi-modality driver action recognition.