# MaskFi: Unsupervised Learning of WiFi and Vision Representations for Multimodal Human Activity Recognition

arXiv: https://arxiv.org/abs/2402.19258 · HTML: https://mlsanity.com/paper/2402.19258/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 43 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers](https://mlsanity.com/paper/2004.00849/) (2020)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2001.07416/) (2020)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing](https://mlsanity.com/paper/2305.10345/) (2023)
- [SenseFi: A Library and Benchmark on Deep-Learning-Empowered WiFi Human Sensing](https://mlsanity.com/paper/2207.07859/) (2022)
- [Joint Activity Recognition and Indoor Localization with WiFi Fingerprints](https://mlsanity.com/paper/1904.04964/) (2019)
- [AirFi: Empowering WiFi-based Passive Human Gesture Recognition to Unseen Environment via Domain Generalization](https://mlsanity.com/paper/2209.10285/) (2022)
- [SiFall: Practical Online Fall Detection with RF Sensing](https://mlsanity.com/paper/2301.03773/) (2023)
- [GaitFi: Robust Device-Free Human Identification via WiFi and Vision Multimodal Learning](https://mlsanity.com/paper/2208.14326/) (2022)
- [SecureSense: Defending Adversarial Attack for Secure Device-Free Human Activity Recognition](https://mlsanity.com/paper/2204.01560/) (2022)
- [Wi-Motion: A Robust Human Activity Recognition Using WiFi Signals](https://mlsanity.com/paper/1810.11705/) (2018)
- [Going Deeper into Recognizing Actions in Dark Environments: A Comprehensive Benchmark Study](https://mlsanity.com/paper/2202.09545/) (2022)

## Built on by (most cited first, 3 of 3)

- [Context-Aware Predictive Coding: A Representation Learning Framework for WiFi Sensing](https://mlsanity.com/paper/2410.01825/) (2024)
- [CIG-MAE: Cross-Modal Information-Guided Masked Autoencoder for Self-Supervised WiFi Sensing](https://mlsanity.com/paper/2512.04723/) (2025)
- [A Short Overview of Multi-Modal Wi-Fi Sensing](https://mlsanity.com/paper/2505.06682/) (2025)

## Abstract (arXiv)

Human activity recognition (HAR) has been playing an increasingly important role in various domains such as healthcare, security monitoring, and metaverse gaming. Though numerous HAR methods based on computer vision have been developed to show prominent performance, they still suffer from poor robustness in adverse visual conditions in particular low illumination, which motivates WiFi-based HAR to serve as a good complementary modality. Existing solutions using WiFi and vision modalities rely on massive labeled data that are very cumbersome to collect. In this paper, we propose a novel unsupervised multimodal HAR solution, MaskFi, that leverages only unlabeled video and WiFi activity data for model training. We propose a new algorithm, masked WiFi-vision modeling (MI2M), that enables the model to learn cross-modal and single-modal features by predicting the masked sections in representation learning. Benefiting from our unsupervised learning procedure, the network requires only a small amount of annotated data for finetuning and can adapt to the new environment with better performance. We conduct extensive experiments on two WiFi-vision datasets collected in-house, and our method achieves human activity recognition and human identification in terms of both robustness and accuracy.