# MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing

arXiv: https://arxiv.org/abs/2305.10345 · HTML: https://mlsanity.com/paper/2305.10345/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 59 references that are in the map.
- First posted on arXiv 12 May 2023.

## Builds on (most cited first, 30 of 31)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling](https://mlsanity.com/paper/2204.13686/) (2022)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda](https://mlsanity.com/paper/2110.05352/) (2021)
- [mm-Pose: Real-Time Human Skeletal Posture Estimation using mmWave Radars and CNNs](https://mlsanity.com/paper/1911.09592/) (2019)
- [MoVi: A Large Multipurpose Motion and Video Dataset](https://mlsanity.com/paper/2003.01888/) (2020)
- [Temporal Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2103.06669/) (2021)
- [SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments](https://mlsanity.com/paper/2303.09095/) (2023)
- [LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors](https://mlsanity.com/paper/2205.15410/) (2022)
- [Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Driving](https://mlsanity.com/paper/2112.12141/) (2021)
- [mRI: Multi-modal 3D Human Pose Estimation Dataset using mmWave, RGB-D, and Inertial Sensors](https://mlsanity.com/paper/2210.08394/) (2022)
- [A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation](https://mlsanity.com/paper/1906.01028/) (2019)
- [SenseFi: A Library and Benchmark on Deep-Learning-Empowered WiFi Human Sensing](https://mlsanity.com/paper/2207.07859/) (2022)
- [LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR Point Clouds](https://mlsanity.com/paper/2203.14698/) (2022)
- [mmBody Benchmark: 3D Body Reconstruction Dataset and Analysis for Millimeter Wave Radar](https://mlsanity.com/paper/2209.05070/) (2022)
- [Fusing Wearable IMUs with Multi-View Images for Human Pose Estimation: A Geometric Approach](https://mlsanity.com/paper/2003.11163/) (2020)
- [CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions](https://mlsanity.com/paper/2303.17948/) (2023)
- [Transferring Dense Pose to Proximal Animal Classes](https://mlsanity.com/paper/2003.00080/) (2020)
- [Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDAR](https://mlsanity.com/paper/2211.16951/) (2022)
- [MetaFi: Device-Free Pose Estimation via Commodity WiFi for Metaverse Avatar Simulation](https://mlsanity.com/paper/2208.10414/) (2022)
- [Fast and Scalable Human Pose Estimation using mmWave Point Cloud](https://mlsanity.com/paper/2205.00097/) (2022)
- [Sparse Representation for 3D Shape Estimation: A Convex Relaxation Approach](https://mlsanity.com/paper/1509.04309/) (2015)
- [DensePose From WiFi](https://mlsanity.com/paper/2301.00250/) (2022)

## Built on by (most cited first, 30 of 79)

- [TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition](https://mlsanity.com/paper/2311.08245/) (2023)
- [Diffusion Model is a Good Pose Estimator from 3D RF-Vision](https://mlsanity.com/paper/2403.16198/) (2024)
- [MMAUD: A Comprehensive Multi-Modal Anti-UAV Dataset for Modern Miniature Drone Threats](https://mlsanity.com/paper/2402.03706/) (2024)
- [WiMANS: A Benchmark Dataset for WiFi-based Multi-user Activity Sensing](https://mlsanity.com/paper/2402.09430/) (2024)
- [IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor data](https://mlsanity.com/paper/2410.02429/) (2024)
- [X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing](https://mlsanity.com/paper/2410.10167/) (2024)
- [Towards Balanced Multi-Modal Learning in 3D Human Pose Estimation](https://mlsanity.com/paper/2501.05264/) (2025)
- [MMVR: Millimeter-wave Multi-View Radar Dataset and Benchmark for Indoor Perception](https://mlsanity.com/paper/2406.10708/) (2024)
- [XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses](https://mlsanity.com/paper/2501.19034/) (2025)
- [A Survey on Wi-Fi Sensing Generalizability: Taxonomy, Techniques, Datasets, and Future Research Prospects](https://mlsanity.com/paper/2503.08008/) (2025)
- [CSI-Bench: A Large-Scale In-the-Wild Dataset for Multi-task WiFi Sensing](https://mlsanity.com/paper/2505.21866/) (2025)
- [Babel: A Scalable Pre-trained Model for Multi-Modal Sensing via Expandable Modality Alignment](https://mlsanity.com/paper/2407.17777/) (2024)
- [SensorQA: A Question Answering Benchmark for Daily-Life Monitoring](https://mlsanity.com/paper/2501.04974/) (2025)
- [RadarLLM: Empowering Large Language Models to Understand Human Motion from Millimeter-Wave Point Cloud Sequence](https://mlsanity.com/paper/2504.09862/) (2025)
- [M4Human: A Large-Scale Multimodal mmWave Radar Benchmark for Human Mesh Reconstruction](https://mlsanity.com/paper/2512.12378/) (2025)
- [RETR: Multi-View Radar Detection Transformer for Indoor Perception](https://mlsanity.com/paper/2411.10293/) (2024)
- [Towards Robust and Realistic Human Pose Estimation via WiFi Signals](https://mlsanity.com/paper/2501.09411/) (2025)
- [Leveraging Foundation Models for Zero-Shot IoT Sensing](https://mlsanity.com/paper/2407.19893/) (2024)
- [Context-Aware Predictive Coding: A Representation Learning Framework for WiFi Sensing](https://mlsanity.com/paper/2410.01825/) (2024)
- [ConSense: Continually Sensing Human Activity with WiFi via Growing and Picking](https://mlsanity.com/paper/2502.17483/) (2025)
- [Graph-based 3D Human Pose Estimation using WiFi Signals](https://mlsanity.com/paper/2511.19105/) (2025)
- [Jacquard V2: Refining Datasets using the Human In the Loop Data Correction Method](https://mlsanity.com/paper/2402.05747/) (2024)
- [MaskFi: Unsupervised Learning of WiFi and Vision Representations for Multimodal Human Activity Recognition](https://mlsanity.com/paper/2402.19258/) (2024)
- [OMuSense-23: A Multimodal Dataset for Contactless Breathing Pattern Recognition and Biometric Analysis](https://mlsanity.com/paper/2407.06137/) (2024)
- [APML: Adaptive Probabilistic Matching Loss for Robust 3D Point Cloud Reconstruction](https://mlsanity.com/paper/2509.08104/) (2025)
- [mmPred: Radar-based Human Motion Prediction in the Dark](https://mlsanity.com/paper/2512.00345/) (2025)
- [Spatio-Temporal 3D Point Clouds from WiFi-CSI Data via Transformer Networks](https://mlsanity.com/paper/2410.16303/) (2024)
- [Acoustic-based 3D Human Pose Estimation Robust to Human Position](https://mlsanity.com/paper/2411.07165/) (2024)
- [DATTA: Domain-Adversarial Test-Time Adaptation for Cross-Domain WiFi-Based Human Activity Recognition](https://mlsanity.com/paper/2411.13284/) (2024)
- [RAPTR: Radar-based 3D Pose Estimation using Transformer](https://mlsanity.com/paper/2511.08387/) (2025)

## Abstract (arXiv)

4D human perception plays an essential role in a myriad of applications, such as home automation and metaverse avatar simulation. However, existing solutions which mainly rely on cameras and wearable devices are either privacy intrusive or inconvenient to use. To address these issues, wireless sensing has emerged as a promising alternative, leveraging LiDAR, mmWave radar, and WiFi signals for device-free human sensing. In this paper, we propose MM-Fi, the first multi-modal non-intrusive 4D human dataset with 27 daily or rehabilitation action categories, to bridge the gap between wireless sensing and high-level human perception tasks. MM-Fi consists of over 320k synchronized frames of five modalities from 40 human subjects. Various annotations are provided to support potential sensing tasks, e.g., human pose estimation and action recognition. Extensive experiments have been conducted to compare the sensing capacity of each or several modalities in terms of multiple tasks. We envision that MM-Fi can contribute to wireless sensing research with respect to action recognition, human pose estimation, multi-modal learning, cross-modal supervision, and interdisciplinary healthcare research.