# Deep Non-rigid Structure-from-Motion: A Sequence-to-Sequence Translation Perspective

arXiv: https://arxiv.org/abs/2204.04730 · HTML: https://mlsanity.com/paper/2204.04730/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 69 references that are in the map.
- First posted on arXiv 10 Apr 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [cosFormer: Rethinking Softmax in Attention](https://mlsanity.com/paper/2202.08791/) (2022)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition](https://mlsanity.com/paper/1912.05534/) (2019)
- [LASR: Learning Articulated Shape Reconstruction from a Monocular Video](https://mlsanity.com/paper/2105.02976/) (2021)
- [Object Relational Graph with Teacher-Recommended Learning for Video Captioning](https://mlsanity.com/paper/2002.11566/) (2020)
- [Deep Subspace Clustering Networks](https://mlsanity.com/paper/1709.02508/) (2017)
- [C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion](https://mlsanity.com/paper/1909.02533/) (2019)
- [Neural Collaborative Subspace Clustering](https://mlsanity.com/paper/1904.10596/) (2019)
- [Unsupervised Adversarial Learning of 3D Human Pose from 2D Joint Locations](https://mlsanity.com/paper/1803.08244/) (2018)
- [Multi-body Non-rigid Structure-from-Motion](https://mlsanity.com/paper/1607.04515/) (2016)
- [Procrustean Regression Networks: Learning 3D Structure of Non-Rigid Objects from 2D Annotations](https://mlsanity.com/paper/2007.10961/) (2020)
- [Deep NRSfM++: Towards Unsupervised 2D-3D Lifting in the Wild](https://mlsanity.com/paper/2001.10090/) (2020)
- [Invariant Teacher and Equivariant Student for Unsupervised 3D Human Pose Estimation](https://mlsanity.com/paper/2012.09398/) (2020)
- [PAUL: Procrustean Autoencoder for Unsupervised Lifting](https://mlsanity.com/paper/2103.16773/) (2021)
- [PR-RRN: Pairwise-Regularized Residual-Recursive Networks for Non-rigid Structure-from-Motion](https://mlsanity.com/paper/2108.07506/) (2021)
- [Dense Non-rigid Structure-from-Motion Made Easy - A Spatial-Temporal Smoothness based Solution](https://mlsanity.com/paper/1706.08629/) (2017)

## Built on by (most cited first, 8 of 8)

- [Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion Modeling](https://mlsanity.com/paper/2308.10705/) (2023)
- [Linear Video Transformer with Feature Fixation](https://mlsanity.com/paper/2210.08164/) (2022)
- [Deep Non-rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling](https://mlsanity.com/paper/2412.07230/) (2024)
- [Non-Rigid Structure-from-Motion via Differential Geometry with Recoverable Conformal Scale](https://mlsanity.com/paper/2510.01665/) (2025)
- [Non-rigid Structure-from-Motion: Temporally-smooth Procrustean Alignment and Spatially-variant Deformation Modeling](https://mlsanity.com/paper/2405.04309/) (2024)
- [Unsupervised 2D-3D lifting of non-rigid objects using local constraints](https://mlsanity.com/paper/2504.19227/) (2025)
- [Enhancing Stress-Strain Predictions with Seq2Seq and Cross-Attention based on Small Punch Test](https://mlsanity.com/paper/2506.17680/) (2025)
- [SSR: A Training-Free Approach for Streaming 3D Reconstruction](https://mlsanity.com/paper/2603.14765/) (2026)

## Abstract (arXiv)

Directly regressing the non-rigid shape and camera pose from the individual 2D frame is ill-suited to the Non-Rigid Structure-from-Motion (NRSfM) problem. This frame-by-frame 3D reconstruction pipeline overlooks the inherent spatial-temporal nature of NRSfM, i.e., reconstructing the whole 3D sequence from the input 2D sequence. In this paper, we propose to model deep NRSfM from a sequence-to-sequence translation perspective, where the input 2D frame sequence is taken as a whole to reconstruct the deforming 3D non-rigid shape sequence. First, we apply a shape-motion predictor to estimate the initial non-rigid shape and camera motion from a single frame. Then we propose a context modeling module to model camera motions and complex non-rigid shapes. To tackle the difficulty in enforcing the global structure constraint within the deep framework, we propose to impose the union-of-subspace structure by replacing the self-expressiveness layer with multi-head attention and delayed regularizers, which enables end-to-end batch-wise training. Experimental results across different datasets such as Human3.6M, CMU Mocap and InterHand prove the superiority of our framework.