# Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos

arXiv: https://arxiv.org/abs/1603.07120 · HTML: https://mlsanity.com/paper/1603.07120/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2016.

## Built on by (most cited first, 26 of 26)

- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [ReconBoost: Boosting Can Achieve Modality Reconcilement](https://mlsanity.com/paper/2405.09321/) (2024)
- [Dynamic Multimodal Fusion](https://mlsanity.com/paper/2204.00102/) (2022)
- [Trear: Transformer-based RGB-D Egocentric Action Recognition](https://mlsanity.com/paper/2101.03904/) (2021)
- [Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble](https://mlsanity.com/paper/2110.06161/) (2021)
- [Towards Robust Multimodal Prompting With Missing Modalities](https://mlsanity.com/paper/2312.15890/) (2023)
- [Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognition](https://mlsanity.com/paper/2112.09129/) (2021)
- [Modality Mixer for Multi-modal Action Recognition](https://mlsanity.com/paper/2208.11314/) (2022)
- [A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition](https://mlsanity.com/paper/2211.09146/) (2022)
- [From CNNs to Transformers in Multimodal Human Action Recognition: A Survey](https://mlsanity.com/paper/2405.15813/) (2024)
- [Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognition](https://mlsanity.com/paper/2202.11374/) (2022)
- [MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset](https://mlsanity.com/paper/2410.19488/) (2024)
- [HighlightMe: Detecting Highlights from Human-Centric Videos](https://mlsanity.com/paper/2110.01774/) (2021)
- [FT-HID: A Large Scale RGB-D Dataset for First and Third Person Human Interaction Analysis](https://mlsanity.com/paper/2209.10155/) (2022)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Extended multi-stream temporal-attention module for skeleton-based human action recognition (HAR)](https://mlsanity.com/paper/2411.06553/) (2024)
- [BST: Badminton Stroke-type Transformer for Skeleton-based Action Recognition in Racket Sports](https://mlsanity.com/paper/2502.21085/) (2025)
- [Cross-view Action Recognition via Contrastive View-invariant Representation](https://mlsanity.com/paper/2305.01733/) (2023)
- [Egocentric RGB+Depth Action Recognition in Industry-Like Settings](https://mlsanity.com/paper/2309.13962/) (2023)
- [Position and Orientation-Aware One-Shot Learning for Medical Action Recognition from Signal Data](https://mlsanity.com/paper/2309.15635/) (2023)
- [A Survey on Human Action Recognition](https://mlsanity.com/paper/2301.06082/) (2022)
- [Learning Scene Flow With Skeleton Guidance For 3D Action Recognition](https://mlsanity.com/paper/2306.13285/) (2023)
- [Unified Contrastive Fusion Transformer for Multimodal Human Action Recognition](https://mlsanity.com/paper/2309.05032/) (2023)
- [Modality Mixer Exploiting Complementary Information for Multi-modal Action Recognition](https://mlsanity.com/paper/2311.12344/) (2023)
- [Modality-Collaborative Low-Rank Decomposers for Few-Shot Video Domain Adaptation](https://mlsanity.com/paper/2511.18711/) (2025)

## Abstract (arXiv)

Single modality action recognition on RGB or depth sequences has been extensively explored recently. It is generally accepted that each of these two modalities has different strengths and limitations for the task of action recognition. Therefore, analysis of the RGB+D videos can help us to better study the complementary properties of these two types of modalities and achieve higher levels of performance. In this paper, we propose a new deep autoencoder based shared-specific feature factorization network to separate input multimodal signals into a hierarchy of components. Further, based on the structure of the features, a structured sparsity learning machine is proposed which utilizes mixed norms to apply regularization within components and group selection between them for better classification performance. Our experimental results show the effectiveness of our cross-modality feature analysis framework by achieving state-of-the-art accuracy for action classification on five challenging benchmark datasets.