# SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesis

arXiv: https://arxiv.org/abs/2204.10211 · HTML: https://mlsanity.com/paper/2204.10211/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 101 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 30 of 46)

- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [Learning to Reconstruct People in Clothing from a Single RGB Camera](https://mlsanity.com/paper/1903.05885/) (2019)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [Learning the Depths of Moving People by Watching Frozen People](https://mlsanity.com/paper/1904.11111/) (2019)
- [Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars](https://mlsanity.com/paper/2008.10174/) (2020)
- [The TUM VI Benchmark for Evaluating Visual-Inertial Odometry](https://mlsanity.com/paper/1804.06120/) (2018)
- [DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor](https://mlsanity.com/paper/1804.06023/) (2018)
- [Detailed, accurate, human shape estimation from clothed 3D scan sequences](https://mlsanity.com/paper/1703.04454/) (2017)
- [SIZER: A Dataset and Model for Parsing 3D Clothing and Learning Size Sensitive 3D Clothing](https://mlsanity.com/paper/2007.11610/) (2020)
- [Driving-Signal Aware Full-Body Avatars](https://mlsanity.com/paper/2105.10441/) (2021)

## Built on by (most cited first, 7 of 7)

- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [LightIt: Illumination Modeling and Control for Diffusion Models](https://mlsanity.com/paper/2403.10615/) (2024)
- [NeuWigs: A Neural Dynamic Model for Volumetric Hair Capture and Animation](https://mlsanity.com/paper/2212.00613/) (2022)
- [CorresNeRF: Image Correspondence Priors for Neural Radiance Fields](https://mlsanity.com/paper/2312.06642/) (2023)
- [Multi-NeuS: 3D Head Portraits from Single Image with Neural Implicit Functions](https://mlsanity.com/paper/2209.04436/) (2022)
- [A Local Appearance Model for Volumetric Capture of Diverse Hairstyle](https://mlsanity.com/paper/2312.08679/) (2023)
- [OmniPoint: Universal Monocular Metric Pointcloud from Any Camera](https://mlsanity.com/paper/2609.09394/) (2026)

## Abstract (arXiv)

We present a dataset of 1000 video sequences of human portraits recorded in real and uncontrolled conditions by using a handheld smartphone accompanied by an external high-quality depth camera. The collected dataset contains 200 people captured in different poses and locations and its main purpose is to bridge the gap between raw measurements obtained from a smartphone and downstream applications, such as state estimation, 3D reconstruction, view synthesis, etc. The sensors employed in data collection are the smartphone's camera and Inertial Measurement Unit (IMU), and an external Azure Kinect DK depth camera software synchronized with sub-millisecond precision to the smartphone system. During the recording, the smartphone flash is used to provide a periodic secondary source of lightning. Accurate mask of the foremost person is provided as well as its impact on the camera alignment accuracy. For evaluation purposes, we compare multiple state-of-the-art camera alignment methods by using a Motion Capture system. We provide a smartphone visual-inertial benchmark for portrait capturing, where we report results for multiple methods and motivate further use of the provided trajectories, available in the dataset, in view synthesis and 3D reconstruction tasks.