# Neural Scene Decomposition for Multi-Person Motion Capture

arXiv: https://arxiv.org/abs/1903.05684 · HTML: https://mlsanity.com/paper/1903.05684/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2019.

## Built on by (most cited first, 21 of 21)

- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation](https://mlsanity.com/paper/2105.02465/) (2021)
- [ElePose: Unsupervised 3D Human Pose Estimation by Predicting Camera Elevation and Learning Normalizing Flows on 2D Poses](https://mlsanity.com/paper/2112.07088/) (2021)
- [GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation](https://mlsanity.com/paper/2112.01036/) (2021)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [Rethinking Self-Supervised Visual Representation Learning in Pre-training for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2303.05370/) (2023)
- [Self-Correctable and Adaptable Inference for Generalizable Human Pose Estimation](https://mlsanity.com/paper/2303.11180/) (2023)
- [Few-shot Geometry-Aware Keypoint Localization](https://mlsanity.com/paper/2303.17216/) (2023)
- [LatentKeypointGAN: Controlling Images via Latent Keypoints](https://mlsanity.com/paper/2103.15812/) (2021)
- [Self-Constrained Inference Optimization on Structural Groups for Human Pose Estimation](https://mlsanity.com/paper/2207.02425/) (2022)
- [Disentangling Geometric Deformation Spaces in Generative Latent Shape Models](https://mlsanity.com/paper/2103.00142/) (2021)
- [Equine Pain Behavior Classification via Self-Supervised Disentangled Pose Representation](https://mlsanity.com/paper/2108.13258/) (2021)
- [Unsupervised 3D Keypoint Discovery with Multi-View Geometry](https://mlsanity.com/paper/2211.12829/) (2022)
- [Understanding Pose and Appearance Disentanglement in 3D Human Pose Estimation](https://mlsanity.com/paper/2309.11667/) (2023)
- [Mask as Supervision: Leveraging Unified Mask Information for Unsupervised 3D Pose Estimation](https://mlsanity.com/paper/2312.07051/) (2023)
- [DifAttack++: Query-Efficient Black-Box Adversarial Attack via Hierarchical Disentangled Feature Space in Cross-Domain](https://mlsanity.com/paper/2406.03017/) (2024)
- [S-AT GCN: Spatial-Attention Graph Convolution Network based Feature Enhancement for 3D Object Detection](https://mlsanity.com/paper/2103.08439/) (2021)
- [DreamTexture: Shape from Virtual Texture with Analysis by Augmentation](https://mlsanity.com/paper/2503.16412/) (2025)
- [LatentKeypointGAN: Controlling Images via Latent Keypoints -- Extended Abstract](https://mlsanity.com/paper/2205.03448/) (2022)
- [Mirror-Aware Neural Humans](https://mlsanity.com/paper/2309.04750/) (2023)
- [X as Supervision: Contending with Depth Ambiguity in Unsupervised Monocular 3D Pose Estimation](https://mlsanity.com/paper/2411.13026/) (2024)

## Abstract (arXiv)

Learning general image representations has proven key to the success of many computer vision tasks. For example, many approaches to image understanding problems rely on deep networks that were initially trained on ImageNet, mostly because the learned features are a valuable starting point to learn from limited labeled data. However, when it comes to 3D motion capture of multiple people, these features are only of limited use. In this paper, we therefore propose an approach to learning features that are useful for this purpose. To this end, we introduce a self-supervised approach to learning what we call a neural scene decomposition (NSD) that can be exploited for 3D pose estimation. NSD comprises three layers of abstraction to represent human subjects: spatial layout in terms of bounding-boxes and relative depth; a 2D shape representation in terms of an instance segmentation mask; and subject-specific appearance and 3D pose information. By exploiting self-supervision coming from multiview data, our NSD model can be trained end-to-end without any 2D or 3D supervision. In contrast to previous approaches, it works for multiple persons and full-frame images. Because it encodes 3D geometry, NSD can then be effectively leveraged to train a 3D pose estimation network from small amounts of annotated data.