# Inertial Hallucinations -- When Wearable Inertial Devices Start Seeing Things

arXiv: https://arxiv.org/abs/2207.06789 · HTML: https://mlsanity.com/paper/2207.06789/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [CentralNet: a Multilayer Approach for Multimodal Fusion](https://mlsanity.com/paper/1808.07275/) (2018)
- [Learning with privileged information via adversarial discriminative modality distillation](https://mlsanity.com/paper/1810.08437/) (2018)
- [HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithm](https://mlsanity.com/paper/2008.01148/) (2020)
- [Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors](https://mlsanity.com/paper/2012.00201/) (2020)
- [Vision and Inertial Sensing Fusion for Human Action Recognition : A Review](https://mlsanity.com/paper/2008.00380/) (2020)
- [Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial Sensors](https://mlsanity.com/paper/1910.11482/) (2019)
- [CaloriNet: From silhouettes to calorie estimation in private environments](https://mlsanity.com/paper/1806.08152/) (2018)
- [Probabilistic Sensor Fusion for Ambient Assisted Living](https://mlsanity.com/paper/1702.01209/) (2017)

## Built on by (most cited first, 1 of 1)

- [Proposal-based Temporal Action Localization with Point-level Supervision](https://mlsanity.com/paper/2310.05511/) (2023)

## Abstract (arXiv)

We propose a novel approach to multimodal sensor fusion for Ambient Assisted Living (AAL) which takes advantage of learning using privileged information (LUPI). We address two major shortcomings of standard multimodal approaches, limited area coverage and reduced reliability. Our new framework fuses the concept of modality hallucination with triplet learning to train a model with different modalities to handle missing sensors at inference time. We evaluate the proposed model on inertial data from a wearable accelerometer device, using RGB videos and skeletons as privileged modalities, and show an improvement of accuracy of an average 6.6% on the UTD-MHAD dataset and an average 5.5% on the Berkeley MHAD dataset, reaching a new state-of-the-art for inertial-only classification accuracy on these datasets. We validate our framework through several ablation studies.