# Towards Adaptive Fusion of Multimodal Deep Networks for Human Action Recognition

arXiv: https://arxiv.org/abs/2512.04943 · HTML: https://mlsanity.com/paper/2512.04943/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 66 references that are in the map.
- First posted on arXiv 4 Dec 2025.

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [Multimodal Intelligence: Representation Learning, Information Fusion, and Applications](https://mlsanity.com/paper/1911.03977/) (2019)
- [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019)
- [Ambient Sound Provides Supervision for Visual Learning](https://mlsanity.com/paper/1608.07017/) (2016)
- [Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera](https://mlsanity.com/paper/1803.05959/) (2018)
- [RWF-2000: An Open Large Scale Video Database for Violence Detection](https://mlsanity.com/paper/1911.05913/) (2019)
- [Learning to Detect Violent Videos using Convolutional Long Short-Term Memory](https://mlsanity.com/paper/1709.06531/) (2017)
- [Learning from THEODORE: A Synthetic Omnidirectional Top-View Indoor Dataset for Deep Transfer Learning](https://mlsanity.com/paper/2011.05719/) (2020)
- [Large Area 3D Human Pose Detection Via Stereo Reconstruction in Panoramic Cameras](https://mlsanity.com/paper/1907.00534/) (2019)
- [Applications of Deep Learning for Top-View Omnidirectional Imaging: A Survey](https://mlsanity.com/paper/2304.08193/) (2023)
- [Gated Self-supervised Learning For Improving Supervised Learning](https://mlsanity.com/paper/2301.05865/) (2023)

## Abstract (arXiv)

This study introduces a pioneering methodology for human action recognition by harnessing deep neural network techniques and adaptive fusion strategies across multiple modalities, including RGB, optical flows, audio, and depth information. Employing gating mechanisms for multimodal fusion, we aim to surpass limitations inherent in traditional unimodal recognition methods while exploring novel possibilities for diverse applications. Through an exhaustive investigation of gating mechanisms and adaptive weighting-based fusion architectures, our methodology enables the selective integration of relevant information from various modalities, thereby bolstering both accuracy and robustness in action recognition tasks. We meticulously examine various gated fusion strategies to pinpoint the most effective approach for multimodal action recognition, showcasing its superiority over conventional unimodal methods. Gating mechanisms facilitate the extraction of pivotal features, resulting in a more holistic representation of actions and substantial enhancements in recognition performance. Our evaluations across human action recognition, violence action detection, and multiple self-supervised learning tasks on benchmark datasets demonstrate promising advancements in accuracy. The significance of this research lies in its potential to revolutionize action recognition systems across diverse fields. The fusion of multimodal information promises sophisticated applications in surveillance and human-computer interaction, especially in contexts related to active assisted living.