# ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions

arXiv: https://arxiv.org/abs/2210.01346 · HTML: https://mlsanity.com/paper/2210.01346/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2011.04841/) (2020)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes](https://mlsanity.com/paper/2001.10692/) (2020)
- [Geometry-based Distance Decomposition for Monocular 3D Object Detection](https://mlsanity.com/paper/2104.03775/) (2021)
- [Human Mesh Recovery from Multiple Shots](https://mlsanity.com/paper/2012.09843/) (2020)
- [Radar-Camera Pixel Depth Association for Depth Completion](https://mlsanity.com/paper/2106.02778/) (2021)
- [Learning 3D Human Shape and Pose from Dense Body Parts](https://mlsanity.com/paper/1912.13344/) (2019)
- [Distant Vehicle Detection Using Radar and Vision](https://mlsanity.com/paper/1901.10951/) (2019)
- [CAT-Det: Contrastively Augmented Transformer for Multi-modal 3D Object Detection](https://mlsanity.com/paper/2204.00325/) (2022)
- [mmBody Benchmark: 3D Body Reconstruction Dataset and Analysis for Millimeter Wave Radar](https://mlsanity.com/paper/2209.05070/) (2022)
- [Rethinking of Radar's Role: A Camera-Radar Dataset and Systematic Annotator via Coordinate Alignment](https://mlsanity.com/paper/2105.05207/) (2021)

## Built on by (most cited first, 21 of 21)

- [RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method](https://mlsanity.com/paper/2403.19501/) (2024)
- [X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing](https://mlsanity.com/paper/2410.10167/) (2024)
- [Towards Balanced Multi-Modal Learning in 3D Human Pose Estimation](https://mlsanity.com/paper/2501.05264/) (2025)
- [Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey](https://mlsanity.com/paper/2410.19872/) (2024)
- [MMBaT: A Multi-task Framework for mmWave-based Human Body Reconstruction and Translation Prediction](https://mlsanity.com/paper/2312.10346/) (2023)
- [mmDEAR: mmWave Point Cloud Density Enhancement for Accurate Human Body Reconstruction](https://mlsanity.com/paper/2503.02375/) (2025)
- [Neural Rendering and Its Hardware Acceleration: A Review](https://mlsanity.com/paper/2402.00028/) (2024)
- [4D Radar Semantic Segmentation of People in Field Conditions Using Temporal Multi-View Networks](https://mlsanity.com/paper/2404.05307/) (2024)
- [ClimbingCap: Multi-Modal Dataset and Method for Rock Climbing in World Coordinate](https://mlsanity.com/paper/2503.21268/) (2025)
- [mmEgoHand: Egocentric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMU](https://mlsanity.com/paper/2501.13805/) (2025)
- [AdaptiveFusion: Adaptive Multi-Modal Multi-View Fusion for 3D Human Body Reconstruction](https://mlsanity.com/paper/2409.04851/) (2024)
- [MAROON: A Dataset for the Joint Characterization of Near-Field High-Resolution Radio-Frequency and Optical Depth Imaging Techniques](https://mlsanity.com/paper/2411.00527/) (2024)
- [Synthesizing and Identifying Noise Levels in Autonomous Vehicle Camera Radar Datasets](https://mlsanity.com/paper/2505.00584/) (2025)
- [HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning](https://mlsanity.com/paper/2505.17645/) (2025)
- [FlashCap: Millisecond-Accurate Human Motion Capture via Flashing LEDs and Event-Based Vision](https://mlsanity.com/paper/2603.19770/) (2026)
- [A Two-Stage Motion-Aware Framework for mmWave-based Human Mesh Recovery](https://mlsanity.com/paper/2605.08530/) (2026)
- [Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation](https://mlsanity.com/paper/2607.18875/) (2026)
- [3D Face Reconstruction From Radar Images](https://mlsanity.com/paper/2412.02403/) (2024)
- [Wave-Former: Through-Occlusion 3D Reconstruction via Wireless Shape Completion](https://mlsanity.com/paper/2511.14152/) (2025)
- [UMEDA: Unified Multi-modal Efficient Data Fusion for Privacy-Preserving Graph Federated Learning via Spectral-Gated Attention and Diffusion-Based Operator Alignment](https://mlsanity.com/paper/2605.08288/) (2026)
- [Contrastive Multi-Modal Hypergraph Reasoning for 3D Crowd Mesh Recovery](https://mlsanity.com/paper/2605.13854/) (2026)

## Abstract (arXiv)

3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather. Complementary, mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather. However, combining RGB and mmWave signals for robust all-weather 3D human reconstruction is still an open challenge, given the sparse nature of mmWave and the vulnerability of RGB images. In this paper, we present ImmFusion, the first mmWave-RGB fusion solution to reconstruct 3D human bodies in all weather conditions robustly. Specifically, our ImmFusion consists of image and point backbones for token feature extraction and a Transformer module for token fusion. The image and point backbones refine global and local features from original data, and the Fusion Transformer Module aims for effective information fusion of two modalities by dynamically selecting informative tokens. Extensive experiments on a large-scale dataset, mmBody, captured in various environments demonstrate that ImmFusion can efficiently utilize the information of two modalities to achieve a robust 3D human body reconstruction in all weather conditions. In addition, our method's accuracy is significantly superior to that of state-of-the-art Transformer-based LiDAR-camera fusion methods.