# Unsupervised 2D-3D lifting of non-rigid objects using local constraints

arXiv: https://arxiv.org/abs/2504.19227 · HTML: https://mlsanity.com/paper/2504.19227/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 27 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017)
- [4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface](https://mlsanity.com/paper/2105.01905/) (2021)
- [C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion](https://mlsanity.com/paper/1909.02533/) (2019)
- [Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet](https://mlsanity.com/paper/2105.02723/) (2021)
- [Deep Non-Rigid Structure from Motion](https://mlsanity.com/paper/1908.00052/) (2019)
- [Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited](https://mlsanity.com/paper/1902.10274/) (2019)
- [Deep Non-rigid Structure-from-Motion: A Sequence-to-Sequence Translation Perspective](https://mlsanity.com/paper/2204.04730/) (2022)
- [MHR-Net: Multiple-Hypothesis Reconstruction of Non-Rigid Shapes from 2D Views](https://mlsanity.com/paper/2207.09086/) (2022)
- [Real-time volumetric rendering of dynamic humans](https://mlsanity.com/paper/2303.11898/) (2023)
- [Deep Non-rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling](https://mlsanity.com/paper/2412.07230/) (2024)
- [End-to-End Learning of Multi-category 3D Pose and Shape Estimation](https://mlsanity.com/paper/2112.10196/) (2021)

## Abstract (arXiv)

For non-rigid objects, predicting the 3D shape from 2D keypoint observations is ill-posed due to occlusions, and the need to disentangle changes in viewpoint and changes in shape. This challenge has often been addressed by embedding low-rank constraints into specialized models. These models can be hard to train, as they depend on finding a canonical way of aligning observations, before they can learn detailed geometry. These constraints have limited the reconstruction quality. We show that generic, high capacity models, trained with an unsupervised loss, allow for more accurate predicted shapes. In particular, applying low-rank constraints to localized subsets of the full shape allows the high capacity to be suitably constrained. We reduce the state-of-the-art reconstruction error on the S-Up3D dataset by over 70%.