mlsanityData to 14 Oct 2026

Modality Mixer for Multi-modal Action Recognition

arXiv 2208.11314 ↗First posted 24 Aug 2022cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

In multi-modal action recognition, it is important to consider not only the complementary nature of different modalities but also global action content. In this paper, we propose a novel network, named Modality Mixer (M-Mixer) network, to leverage complementary information across modalities and temporal context of an action for multi-modal action recognition. We also introduce a simple yet effective recurrent unit, called Multi-modal Contextualization Unit (MCU), which is a core component of M-Mixer. Our MCU temporally encodes a sequence of one modality (e.g., RGB) with action content features of other modalities (e.g., depth, IR). This process encourages M-Mixer to exploit global action content and also to supplement complementary information of other modalities. As a result, our proposed method outperforms state-of-the-art methods on NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA datasets. Moreover, we demonstrate the effectiveness of M-Mixer by conducting comprehensive ablation studies.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 24 Aug 2022.

Builds on

36 of 43 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  6. 2017Non-local Neural Networkscited by 2,293
  7. 2018SlowFast Networks for Video Recognitioncited by 1,867
  8. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  9. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  10. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  11. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  12. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
24 more
  1. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  2. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  3. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  4. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  5. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  6. 2021Revisiting Skeleton-based Action Recognitioncited by 257
  7. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
  8. 2014Cross-view Action Modeling, Learning and Recognitioncited by 134
  9. 2018Modality Distillation with Multiple Stream Networks for Action Recognitioncited by 83
  10. 2017Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognitioncited by 82
  11. 2018D3D: Distilled 3D Networks for Video Action Recognitioncited by 60
  12. 2020VPN: Learning Video-Pose Embedding for Activities of Daily Livingcited by 57
  13. 2018Representation Flow for Action Recognitioncited by 51
  14. 2021VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Livingcited by 35
  15. 2018Motion Feature Network: Fixed Motion Filter for Action Recognitioncited by 34
  16. 2018Learning with privileged information via adversarial discriminative modality distillationcited by 30
  17. 2019Dance with Flow: Two-in-One Stream Action Detectioncited by 30
  18. 2019DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognitioncited by 30
  19. 2020HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithmcited by 29
  20. 2016Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videoscited by 26
  21. 2019View-invariant Deep Architecture for Human Action Recognition using late fusioncited by 15
  22. 2017Viewpoint Invariant Action Recognition using RGB-D Videoscited by 5
  23. 2017Learning Approximate Stochastic Transition Modelscited by 3
  24. 2018Random Temporal Skipping for Multirate Video Analysiscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 2; 2026: 1.

9 papers.

  1. 2023Audio-Visual Glance Network for Efficient Video Recognitioncited by 4
  2. 2023Multi-stage Factorized Spatio-Temporal Representation for RGB-D Action and Gesture Recognitioncited by 1
  3. 2024Spatio-Temporal Proximity-Aware Dual-Path Model for Panoramic Activity Recognitioncited by 1
  4. 2024Joint-Dataset Learning and Cross-Consistent Regularization for Text-to-Motion Retrievalcited by 1
  5. 2023Sketch-based Video Object Localizationcited by 0
  6. 2023Modality Mixer Exploiting Complementary Information for Multi-modal Action Recognitioncited by 0
  7. 2025Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixupcited by 0
  8. 2025Solution for 8th Competition on Affective & Behavior Analysis in-the-wildcited by 0
  9. 2026Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detectioncited by 0