# Gimme Signals: Discriminative signal encoding for multimodal activity recognition

arXiv: https://arxiv.org/abs/2003.06156 · HTML: https://mlsanity.com/paper/2003.06156/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2020.

## Built on by (most cited first, 18 of 18)

- [Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors](https://mlsanity.com/paper/2109.02993/) (2021)
- [Fusion-GCN: Multimodal Action Recognition using Graph Convolutional Networks](https://mlsanity.com/paper/2109.12946/) (2021)
- [Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition](https://mlsanity.com/paper/2205.10071/) (2022)
- [Multimodal sensor fusion in the latent representation space](https://mlsanity.com/paper/2208.02183/) (2022)
- [Sensor Data Augmentation from Skeleton Pose Sequences for Improving Human Activity Recognition](https://mlsanity.com/paper/2406.16886/) (2024)
- [From CNNs to Transformers in Multimodal Human Action Recognition: A Survey](https://mlsanity.com/paper/2405.15813/) (2024)
- [Self-Supervised Multimodal Fusion Transformer for Passive Activity Recognition](https://mlsanity.com/paper/2209.03765/) (2022)
- [ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization](https://mlsanity.com/paper/2208.09414/) (2022)
- [Temporal Feature Alignment in Contrastive Self-Supervised Learning for Human Activity Recognition](https://mlsanity.com/paper/2210.03382/) (2022)
- [AFE-CNN: 3D Skeleton-based Action Recognition with Action Feature Enhancement](https://mlsanity.com/paper/2208.03444/) (2022)
- [Streamlining Multimodal Data Fusion in Wireless Communication and Sensor Networks](https://mlsanity.com/paper/2302.12636/) (2023)
- [CSI-Based Efficient Self-Quarantine Monitoring System Using Branchy Convolution Neural Network](https://mlsanity.com/paper/2306.01756/) (2023)
- [Non-contact Multimodal Indoor Human Monitoring Systems: A Survey](https://mlsanity.com/paper/2312.07601/) (2023)
- [A Novel Two Stream Decision Level Fusion of Vision and Inertial Sensors Data for Automatic Multimodal Human Activity Recognition System](https://mlsanity.com/paper/2306.15765/) (2023)
- [Component attention network for multimodal dance improvisation recognition](https://mlsanity.com/paper/2310.05938/) (2023)
- [Distilled Mid-Fusion Transformer Networks for Multi-Modal Human Activity Recognition](https://mlsanity.com/paper/2305.03810/) (2023)
- [Unified Contrastive Fusion Transformer for Multimodal Human Action Recognition](https://mlsanity.com/paper/2309.05032/) (2023)
- [Person Segmentation and Action Classification for Multi-Channel Hemisphere Field of View LiDAR Sensors](https://mlsanity.com/paper/2411.11151/) (2024)

## Abstract (arXiv)

We present a simple, yet effective and flexible method for action recognition supporting multiple sensor modalities. Multivariate signal sequences are encoded in an image and are then classified using a recently proposed EfficientNet CNN architecture. Our focus was to find an approach that generalizes well across different sensor modalities without specific adaptions while still achieving good results. We apply our method to 4 action recognition datasets containing skeleton sequences, inertial and motion capturing measurements as well as wifi fingerprints that range up to 120 action classes. Our method defines the current best CNN-based approach on the NTU RGB+D 120 dataset, lifts the state of the art on the ARIL Wi-Fi dataset by +6.78%, improves the UTD-MHAD inertial baseline by +14.4%, the UTD-MHAD skeleton baseline by 1.13% and achieves 96.11% on the Simitate motion capturing data (80/20 split). We further demonstrate experiments on both, modality fusion on a signal level and signal reduction to prevent the representation from overloading.