# HUMAN4D: A Human-Centric Multimodal Dataset for Motions and Immersive Media

arXiv: https://arxiv.org/abs/2110.07235 · HTML: https://mlsanity.com/paper/2110.07235/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 82 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera](https://mlsanity.com/paper/1705.01583/) (2017)
- [Self-Supervised Learning of 3D Human Pose using Multi-view Geometry](https://mlsanity.com/paper/1903.02330/) (2019)
- [LiveCap: Real-time Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1810.02648/) (2018)
- [Cross View Fusion for 3D Human Pose Estimation](https://mlsanity.com/paper/1909.01203/) (2019)
- [A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Image](https://mlsanity.com/paper/1908.09999/) (2019)
- [Learning Convolutional Transforms for Lossy Point Cloud Geometry Compression](https://mlsanity.com/paper/1903.08548/) (2019)
- [3D Human Body Reconstruction from a Single Image via Volumetric Regression](https://mlsanity.com/paper/1809.03770/) (2018)
- [Recurrent Human Pose Estimation](https://mlsanity.com/paper/1605.02914/) (2016)
- [Deep Implicit Volume Compression](https://mlsanity.com/paper/2005.08877/) (2020)
- [A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms](https://mlsanity.com/paper/1701.07372/) (2017)
- [An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing](https://mlsanity.com/paper/1712.03084/) (2017)
- [PoseNet3D: Learning Temporally Consistent 3D Human Pose via Knowledge Distillation](https://mlsanity.com/paper/2003.03473/) (2020)
- [Real-time marker-less multi-person 3D pose estimation in RGB-Depth camera networks](https://mlsanity.com/paper/1710.06235/) (2017)
- [Bottom-up Pose Estimation of Multiple Person with Bounding Box Constraint](https://mlsanity.com/paper/1807.09972/) (2018)
- [HUMBI: A Large Multiview Dataset of Human Body Expressions](https://mlsanity.com/paper/1812.00281/) (2018)
- [A Low-Cost, Flexible and Portable Volumetric Capturing System](https://mlsanity.com/paper/1909.01207/) (2019)

## Built on by (most cited first, 29 of 29)

- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](https://mlsanity.com/paper/2210.06551/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Back to MLP: A Simple Baseline for Human Motion Prediction](https://mlsanity.com/paper/2207.01567/) (2022)
- [BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction](https://mlsanity.com/paper/2211.14304/) (2022)
- [UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture](https://mlsanity.com/paper/2208.01633/) (2022)
- [Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats](https://mlsanity.com/paper/2212.14474/) (2022)
- [FLEX: Full-Body Grasping Without Full-Body Grasps](https://mlsanity.com/paper/2211.11903/) (2022)
- [AnimateAnyMesh: A Feed-Forward 4D Foundation Model for Text-Driven Universal Mesh Animation](https://mlsanity.com/paper/2506.09982/) (2025)
- [Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera](https://mlsanity.com/paper/2401.00847/) (2024)
- [Conditional Motion In-betweening](https://mlsanity.com/paper/2202.04307/) (2022)
- [SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data](https://mlsanity.com/paper/2311.02191/) (2023)
- [4DHumanOutfit: a multi-subject 4D dataset of human motion sequences in varying outfits exhibiting large displacements](https://mlsanity.com/paper/2306.07399/) (2023)
- [Constrained Diffusion with Trust Sampling](https://mlsanity.com/paper/2411.10932/) (2024)
- [Reconstructing Close Human Interactions from Multiple Views](https://mlsanity.com/paper/2401.16173/) (2024)
- [FSVVD: A Dataset of Full Scene Volumetric Video](https://mlsanity.com/paper/2303.03599/) (2023)
- [A Flexible Multi-view Multi-modal Imaging System for Outdoor Scenes](https://mlsanity.com/paper/2302.10465/) (2023)
- [Headset: Human emotion awareness under partial occlusions multimodal dataset](https://mlsanity.com/paper/2402.09107/) (2024)
- [Motion Capture from Inertial and Vision Sensors](https://mlsanity.com/paper/2407.16341/) (2024)
- [MV-TAP: Tracking Any Point in Multi-View Videos](https://mlsanity.com/paper/2512.02006/) (2025)
- [DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimization](https://mlsanity.com/paper/2406.14567/) (2024)
- [NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data](https://mlsanity.com/paper/2603.18496/) (2026)
- [RoMo: A Large-Scale, Richly Organized Dataset and Semantic Taxonomy for Human Motion Generation](https://mlsanity.com/paper/2605.26241/) (2026)
- [Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion](https://mlsanity.com/paper/2406.11196/) (2024)
- [Shape-aware Inertial Poser: Motion Tracking for Humans with Diverse Shapes Using Sparse Inertial Sensors](https://mlsanity.com/paper/2510.17101/) (2025)
- [MMCM: Multimodality-aware Metric using Clustering-based Modes for Probabilistic Human Motion Prediction](https://mlsanity.com/paper/2511.15179/) (2025)
- [VolHuMe: a High-Resolution Large Scale Dataset of Volumetric Human Meshes](https://mlsanity.com/paper/2606.23062/) (2026)
- [On Coordinate Decoding for Keypoint Estimation Tasks](https://mlsanity.com/paper/2110.10289/) (2021)
- [L-SR1: Learned Symmetric-Rank-One Preconditioning](https://mlsanity.com/paper/2508.12270/) (2025)
- [A Dataset and Evaluation for Complex 4D Markerless Human Motion Capture](https://mlsanity.com/paper/2604.12765/) (2026)

## Abstract (arXiv)

We introduce HUMAN4D, a large and multimodal 4D dataset that contains a variety of human activities simultaneously captured by a professional marker-based MoCap, a volumetric capture and an audio recording system. By capturing 2 female and 2 male professional actors performing various full-body movements and expressions, HUMAN4D provides a diverse set of motions and poses encountered as part of single- and multi-person daily, physical and social activities (jumping, dancing, etc.), along with multi-RGBD (mRGBD), volumetric and audio data. Despite the existence of multi-view color datasets captured with the use of hardware (HW) synchronization, to the best of our knowledge, HUMAN4D is the first and only public resource that provides volumetric depth maps with high synchronization precision due to the use of intra- and inter-sensor HW-SYNC. Moreover, a spatio-temporally aligned scanned and rigged 3D character complements HUMAN4D to enable joint research on time-varying and high-quality dynamic meshes. We provide evaluation baselines by benchmarking HUMAN4D with state-of-the-art human pose estimation and 3D compression methods. For the former, we apply 2D and 3D pose estimation algorithms both on single- and multi-view data cues. For the latter, we benchmark open-source 3D codecs on volumetric data respecting online volumetric video encoding and steady bit-rates. Furthermore, qualitative and quantitative visual comparison between mesh-based volumetric data reconstructed in different qualities showcases the available options with respect to 4D representations. HUMAN4D is introduced to the computer vision and graphics research communities to enable joint research on spatio-temporally aligned pose, volumetric, mRGBD and audio data cues. The dataset and its code are available https://tofis.github.io/myurls/human4d.