# Robust Human Registration with Body Part Segmentation on Noisy Point Clouds

arXiv: https://arxiv.org/abs/2504.03602 · HTML: https://mlsanity.com/paper/2504.03602/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Builds on (most cited first, 23 of 23)

- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [Monocular Total Capture: Posing Face, Body, and Hands in the Wild](https://mlsanity.com/paper/1812.01598/) (2018)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction](https://mlsanity.com/paper/2007.11432/) (2020)
- [MeshSDF: Differentiable Iso-Surface Extraction](https://mlsanity.com/paper/2006.03997/) (2020)
- [LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration](https://mlsanity.com/paper/2010.12447/) (2020)
- [InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction](https://mlsanity.com/paper/2209.12354/) (2022)
- [Hi4D: 4D Instance Segmentation of Close Human Interaction](https://mlsanity.com/paper/2303.15380/) (2023)
- [Phantom: Training Robots Without Robots Using Only Human Videos](https://mlsanity.com/paper/2503.00779/) (2025)
- [Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration](https://mlsanity.com/paper/2104.08160/) (2021)
- [3D Segmentation of Humans in Point Clouds with Synthetic Data](https://mlsanity.com/paper/2212.00786/) (2022)
- [NICP: Neural ICP for 3D Human Registration at Scale](https://mlsanity.com/paper/2312.14024/) (2023)
- [Generalizing Neural Human Fitting to Unseen Poses With Articulated SE(3) Equivariance](https://mlsanity.com/paper/2304.10528/) (2023)
- [PointHPS: Cascaded 3D Human Pose and Shape Estimation from Point Clouds](https://mlsanity.com/paper/2308.14492/) (2023)
- [Self-supervised 3D Human Mesh Recovery from Noisy Point Clouds](https://mlsanity.com/paper/2107.07539/) (2021)

## Built on by (most cited first, 2 of 2)

- [OmniFit: Multi-modal 3D Body Fitting via Scale-agnostic Dense Landmark Prediction](https://mlsanity.com/paper/2604.21575/) (2026)
- [Towards Automated Initial Probe Placement in Transthoracic Teleultrasound Using Human Mesh and Skeleton Recovery](https://mlsanity.com/paper/2603.11257/) (2026)

## Abstract (arXiv)

Registering human meshes to 3D point clouds is essential for applications such as augmented reality and human-robot interaction but often yields imprecise results due to noise and background clutter in real-world data. We introduce a hybrid approach that incorporates body-part segmentation into the mesh fitting process, enhancing both human pose estimation and segmentation accuracy. Our method first assigns body part labels to individual points, which then guide a two-step SMPL-X fitting: initial pose and orientation estimation using body part centroids, followed by global refinement of the point cloud alignment. Additionally, we demonstrate that the fitted human mesh can refine body part labels, leading to improved segmentation. Evaluations on the cluttered and noisy real-world datasets InterCap, EgoBody, and BEHAVE show that our approach significantly outperforms prior methods in both pose estimation and segmentation accuracy. Code and results are available on our project website: https://segfit.github.io