# MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization

arXiv: https://arxiv.org/abs/2101.06605 · HTML: https://mlsanity.com/paper/2101.06605/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 90 references that are in the map.
- First posted on arXiv 17 Jan 2021.

## Builds on (most cited first, 30 of 48)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)
- [FLOT: Scene Flow on Point Clouds Guided by Optimal Transport](https://mlsanity.com/paper/2007.11142/) (2020)
- [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019)
- [MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences](https://mlsanity.com/paper/1910.09165/) (2019)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [SE-Sync: A Certifiably Correct Algorithm for Synchronization over the Special Euclidean Group](https://mlsanity.com/paper/1612.07386/) (2016)
- [Learning multiview 3D point cloud registration](https://mlsanity.com/paper/2001.05119/) (2020)
- [Joint Monocular 3D Vehicle Detection and Tracking](https://mlsanity.com/paper/1811.10742/) (2018)
- [Deep Part Induction from Articulated Object Pairs](https://mlsanity.com/paper/1809.07417/) (2018)
- [MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM](https://mlsanity.com/paper/1812.07976/) (2018)
- [FlowNet3D++: Geometric Losses For Deep Scene Flow Estimation](https://mlsanity.com/paper/1912.01438/) (2019)
- [Deep Rigid Instance Scene Flow](https://mlsanity.com/paper/1904.08913/) (2019)
- [CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations](https://mlsanity.com/paper/2008.02792/) (2020)
- [DynaSLAM II: Tightly-Coupled Multi-Object Tracking and SLAM](https://mlsanity.com/paper/2010.07820/) (2020)
- [Self-Supervised Learning of Non-Rigid Residual Flow and Ego-Motion](https://mlsanity.com/paper/2009.10467/) (2020)

## Built on by (most cited first, 30 of 46)

- [Lepard: Learning partial point cloud matching in rigid and deformable scenes](https://mlsanity.com/paper/2111.12591/) (2021)
- [Real2Code: Reconstruct Articulated Objects via Code Generation](https://mlsanity.com/paper/2406.08474/) (2024)
- [OPD: Single-view 3D Openable Part Detection](https://mlsanity.com/paper/2203.16421/) (2022)
- [Dynamic 3D Scene Analysis by Point Cloud Accumulation](https://mlsanity.com/paper/2207.12394/) (2022)
- [Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery](https://mlsanity.com/paper/2105.01047/) (2021)
- [Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling](https://mlsanity.com/paper/2502.02590/) (2025)
- [D3D-HOI: Dynamic 3D Human-Object Interactions from Videos](https://mlsanity.com/paper/2108.08420/) (2021)
- [Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Clouds](https://mlsanity.com/paper/2306.00979/) (2023)
- [Projective Manifold Gradient Layer for Deep Rotation Regression](https://mlsanity.com/paper/2110.11657/) (2021)
- [Multiway Non-rigid Point Cloud Registration via Learned Functional Map Synchronization](https://mlsanity.com/paper/2111.12878/) (2021)
- [DRAWER: Digital Reconstruction and Articulation With Environment Realism](https://mlsanity.com/paper/2504.15278/) (2025)
- [OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds](https://mlsanity.com/paper/2210.04458/) (2022)
- [Category-Independent Articulated Object Tracking with Factor Graphs](https://mlsanity.com/paper/2205.03721/) (2022)
- [S2O: Static to Openable Enhancement for Articulated 3D Objects](https://mlsanity.com/paper/2409.18896/) (2024)
- [Morig: Motion-aware rigging of character meshes from point clouds](https://mlsanity.com/paper/2210.09463/) (2022)
- [Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance](https://mlsanity.com/paper/2305.16314/) (2023)
- [EFEM: Equivariant Neural Field Expectation Maximization for 3D Object Segmentation Without Scene Supervision](https://mlsanity.com/paper/2303.15440/) (2023)
- [Sequential Manipulation Planning on Scene Graph](https://mlsanity.com/paper/2207.04364/) (2022)
- [Quantum Motion Segmentation](https://mlsanity.com/paper/2203.13185/) (2022)
- [Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments](https://mlsanity.com/paper/2312.09138/) (2023)
- [Survey on Modeling of Human-made Articulated Objects](https://mlsanity.com/paper/2403.14937/) (2024)
- [Rectified Point Flow: Generic Point Cloud Pose Estimation](https://mlsanity.com/paper/2506.05282/) (2025)
- [Unsupervised Pose-Aware Part Decomposition for 3D Articulated Objects](https://mlsanity.com/paper/2110.04411/) (2021)
- [Kinematify: Open-Vocabulary Synthesis of High-DoF Articulated Objects](https://mlsanity.com/paper/2511.01294/) (2025)
- [Q-FW: A Hybrid Classical-Quantum Frank-Wolfe for Quadratic Binary Optimization](https://mlsanity.com/paper/2203.12633/) (2022)
- [OP-Align: Object-level and Part-level Alignment for Self-supervised Category-level Articulated Object Pose Estimation](https://mlsanity.com/paper/2408.16547/) (2024)
- [Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation](https://mlsanity.com/paper/2306.05584/) (2023)
- [Active Coarse-to-Fine Segmentation of Moveable Parts from Real Images](https://mlsanity.com/paper/2303.11530/) (2023)
- [Learning Part Motion of Articulated Objects Using Spatially Continuous Neural Implicit Representations](https://mlsanity.com/paper/2311.12407/) (2023)
- [AutoURDF: Unsupervised Robot Modeling from Point Cloud Frames Using Cluster Registration](https://mlsanity.com/paper/2412.05507/) (2024)

## Abstract (arXiv)

We present MultiBodySync, a novel, end-to-end trainable multi-body motion segmentation and rigid registration framework for multiple input 3D point clouds. The two non-trivial challenges posed by this multi-scan multibody setting that we investigate are: (i) guaranteeing correspondence and segmentation consistency across multiple input point clouds capturing different spatial arrangements of bodies or body parts; and (ii) obtaining robust motion-based rigid body segmentation applicable to novel object categories. We propose an approach to address these issues that incorporates spectral synchronization into an iterative deep declarative network, so as to simultaneously recover consistent correspondences as well as motion segmentation. At the same time, by explicitly disentangling the correspondence and motion segmentation estimation modules, we achieve strong generalizability across different object categories. Our extensive evaluations demonstrate that our method is effective on various datasets ranging from rigid parts in articulated objects to individually moving objects in a 3D scene, be it single-view or full point clouds.