# CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognition

arXiv: https://arxiv.org/abs/2406.11340 · HTML: https://mlsanity.com/paper/2406.11340/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [PIVOT: Prompting for Video Continual Learning](https://mlsanity.com/paper/2212.04842/) (2022)
- [TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration](https://mlsanity.com/paper/2203.00927/) (2022)
- [Dynamic Spatio-Temporal Specialization Learning for Fine-Grained Action Recognition](https://mlsanity.com/paper/2209.01425/) (2022)
- [Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition](https://mlsanity.com/paper/2101.06636/) (2021)
- [A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding](https://mlsanity.com/paper/2204.04734/) (2022)
- [Multi-modality action recognition based on dual feature shift in vehicle cabin monitoring](https://mlsanity.com/paper/2401.14838/) (2024)
- [Supervised Contrastive Learning for Detecting Anomalous Driving Behaviours from Multimodal Videos](https://mlsanity.com/paper/2109.04021/) (2021)

## Built on by (most cited first, 6 of 6)

- [ByteNet: Rethinking Multimedia File Fragment Classification through Visual Perspectives](https://mlsanity.com/paper/2410.20855/) (2024)
- [MultiFuser: Multimodal Fusion Transformer for Enhanced Driver Action Recognition](https://mlsanity.com/paper/2408.01766/) (2024)
- [DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Dataset](https://mlsanity.com/paper/2601.11990/) (2026)
- [Mixture-of-Modality-Experts with Holistic Token Learning for Fine-Grained Multimodal Visual Analytics in Driver Action Recognition](https://mlsanity.com/paper/2604.05947/) (2026)
- [From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognition](https://mlsanity.com/paper/2507.14686/) (2025)
- [T-MASK: Temporal Masking for Probing Foundation Models across Camera Views in Driver Monitoring](https://mlsanity.com/paper/2508.16207/) (2025)

## Abstract (arXiv)

Driver action recognition has significantly advanced in enhancing driver-vehicle interactions and ensuring driving safety by integrating multiple modalities, such as infrared and depth. Nevertheless, compared to RGB modality only, it is always laborious and costly to collect extensive data for all types of non-RGB modalities in car cabin environments. Therefore, previous works have suggested independently learning each non-RGB modality by fine-tuning a model pre-trained on RGB videos, but these methods are less effective in extracting informative features when faced with newly-incoming modalities due to large domain gaps. In contrast, we propose a Continual Cross-Modal Mapping Network (CM2-Net) to continually learn each newly-incoming modality with instructive prompts from the previously-learned modalities. Specifically, we have developed Accumulative Cross-modal Mapping Prompting (ACMP), to map the discriminative and informative features learned from previous modalities into the feature space of newly-incoming modalities. Then, when faced with newly-incoming modalities, these mapped features are able to provide effective prompts for which features should be extracted and prioritized. These prompts are accumulating throughout the continual learning process, thereby boosting further recognition performances. Extensive experiments conducted on the Drive&Act dataset demonstrate the performance superiority of CM2-Net on both uni- and multi-modal driver action recognition.