# Robust Multimodal Fusion for Human Activity Recognition

arXiv: https://arxiv.org/abs/2303.04636 · HTML: https://mlsanity.com/paper/2303.04636/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 61 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables](https://mlsanity.com/paper/1604.08880/) (2016)
- [DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing](https://mlsanity.com/paper/1611.01942/) (2016)
- [Variational Recurrent Auto-Encoders](https://mlsanity.com/paper/1412.6581/) (2014)
- [Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding](https://mlsanity.com/paper/2005.09202/) (2020)
- [Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data](https://mlsanity.com/paper/1808.02096/) (2018)
- [Understanding and Improving Recurrent Networks for Human Activity Recognition by Continuous Attention](https://mlsanity.com/paper/1810.04038/) (2018)
- [STFNets: Learning Sensing Signals from the Time-Frequency Perspective with Short-Time Fourier Neural Networks](https://mlsanity.com/paper/1902.07849/) (2019)
- [An Efficient Approach to Informative Feature Extraction from Multimodal Data](https://mlsanity.com/paper/1811.08979/) (2018)
- [Denoising Adversarial Autoencoders](https://mlsanity.com/paper/1703.01220/) (2017)
- [Multi-modal Fusion for Single-Stage Continuous Gesture Recognition](https://mlsanity.com/paper/2011.04945/) (2020)
- [Learning Selective Sensor Fusion for States Estimation](https://mlsanity.com/paper/1912.13077/) (2019)
- [Attention-Based Deep Learning Framework for Human Activity Recognition with User Adaptation](https://mlsanity.com/paper/2006.03820/) (2020)

## Built on by (most cited first, 1 of 1)

- [CNN Autoencoders for Hierarchical Feature Extraction and Fusion in Multi-sensor Human Activity Recognition](https://mlsanity.com/paper/2502.04489/) (2025)

## Abstract (arXiv)

The proliferation of IoT and mobile devices equipped with heterogeneous sensors has enabled new applications that rely on the fusion of time-series data generated by multiple sensors with different modalities. While there are promising deep neural network architectures for multimodal fusion, their performance falls apart quickly in the presence of consecutive missing data and noise across multiple modalities/sensors, the issues that are prevalent in real-world settings. We propose Centaur, a multimodal fusion model for human activity recognition (HAR) that is robust to these data quality issues. Centaur combines a data cleaning module, which is a denoising autoencoder with convolutional layers, and a multimodal fusion module, which is a deep convolutional neural network with the self-attention mechanism to capture cross-sensor correlation. We train Centaur using a stochastic data corruption scheme and evaluate it on three datasets that contain data generated by multiple inertial measurement units. Centaur's data cleaning module outperforms 2 state-of-the-art autoencoder-based models and its multimodal fusion module outperforms 4 strong baselines. Compared to 2 related robust fusion architectures, Centaur is more robust, achieving 11.59-17.52% higher accuracy in HAR, especially in the presence of consecutive missing data in multiple sensor channels.