# Cooperative Training of Deep Aggregation Networks for RGB-D Action Recognition

arXiv: https://arxiv.org/abs/1801.01080 · HTML: https://mlsanity.com/paper/1801.01080/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2017.

## Built on by (most cited first, 11 of 11)

- [Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition](https://mlsanity.com/paper/2112.10992/) (2021)
- [Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognition](https://mlsanity.com/paper/2112.09129/) (2021)
- [Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition](https://mlsanity.com/paper/2102.05348/) (2021)
- [A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition](https://mlsanity.com/paper/2211.09146/) (2022)
- [Video Violence Recognition and Localization Using a Semi-Supervised Hard Attention Model](https://mlsanity.com/paper/2202.02212/) (2022)
- [FT-HID: A Large Scale RGB-D Dataset for First and Third Person Human Interaction Analysis](https://mlsanity.com/paper/2209.10155/) (2022)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Multi-stage Factorized Spatio-Temporal Representation for RGB-D Action and Gesture Recognition](https://mlsanity.com/paper/2308.12006/) (2023)
- [Egocentric RGB+Depth Action Recognition in Industry-Like Settings](https://mlsanity.com/paper/2309.13962/) (2023)
- [Modality Mixer Exploiting Complementary Information for Multi-modal Action Recognition](https://mlsanity.com/paper/2311.12344/) (2023)
- [KRAST: Knowledge-Augmented Robotic Action Recognition with Structured Text for Vision-Language Models](https://mlsanity.com/paper/2509.16452/) (2025)

## Abstract (arXiv)

A novel deep neural network training paradigm that exploits the conjoint information in multiple heterogeneous sources is proposed. Specifically, in a RGB-D based action recognition task, it cooperatively trains a single convolutional neural network (named c-ConvNet) on both RGB visual features and depth features, and deeply aggregates the two kinds of features for action recognition. Differently from the conventional ConvNet that learns the deep separable features for homogeneous modality-based classification with only one softmax loss function, the c-ConvNet enhances the discriminative power of the deeply learned features and weakens the undesired modality discrepancy by jointly optimizing a ranking loss and a softmax loss for both homogeneous and heterogeneous modalities. The ranking loss consists of intra-modality and cross-modality triplet losses, and it reduces both the intra-modality and cross-modality feature variations. Furthermore, the correlations between RGB and depth data are embedded in the c-ConvNet, and can be retrieved by either of the modalities and contribute to the recognition in the case even only one of the modalities is available. The proposed method was extensively evaluated on two large RGB-D action recognition datasets, ChaLearn LAP IsoGD and NTU RGB+D datasets, and one small dataset, SYSU 3D HOI, and achieved state-of-the-art results.