mlsanityData to 7 Oct 2026

Deep Non-rigid Structure-from-Motion: A Sequence-to-Sequence Translation Perspective

arXiv 2204.04730 ↗First posted 10 Apr 2022cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Directly regressing the non-rigid shape and camera pose from the individual 2D frame is ill-suited to the Non-Rigid Structure-from-Motion (NRSfM) problem. This frame-by-frame 3D reconstruction pipeline overlooks the inherent spatial-temporal nature of NRSfM, i.e., reconstructing the whole 3D sequence from the input 2D sequence. In this paper, we propose to model deep NRSfM from a sequence-to-sequence translation perspective, where the input 2D frame sequence is taken as a whole to reconstruct the deforming 3D non-rigid shape sequence. First, we apply a shape-motion predictor to estimate the initial non-rigid shape and camera motion from a single frame. Then we propose a context modeling module to model camera motions and complex non-rigid shapes. To tackle the difficulty in enforcing the global structure constraint within the deep framework, we propose to impose the union-of-subspace structure by replacing the self-expressiveness layer with multi-head attention and delayed regularizers, which enables end-to-end batch-wise training. Experimental results across different datasets such as Human3.6M, CMU Mocap and InterHand prove the superiority of our framework.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 31 of its 69 references that are in the map. First posted on arXiv 10 Apr 2022.

Builds on

31 of 69 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017Mask R-CNNcited by 7,259
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2018YOLOv3: An Incremental Improvementcited by 2,652
  9. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  10. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  12. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
19 more
  1. 2018Learning 3D Human Dynamics from Videocited by 315
  2. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  3. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  4. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 195
  5. 2019Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognitioncited by 117
  6. 2021LASR: Learning Articulated Shape Reconstruction from a Monocular Videocited by 115
  7. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 106
  8. 2017Deep Subspace Clustering Networkscited by 94
  9. 2019C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motioncited by 60
  10. 2019Neural Collaborative Subspace Clusteringcited by 21
  11. 2018Unsupervised Adversarial Learning of 3D Human Pose from 2D Joint Locationscited by 18
  12. 2016Multi-body Non-rigid Structure-from-Motioncited by 13
  13. 2020Procrustean Regression Networks: Learning 3D Structure of Non-Rigid Objects from 2D Annotationscited by 13
  14. 2020Deep NRSfM++: Towards Unsupervised 2D-3D Lifting in the Wildcited by 10
  15. 2020Invariant Teacher and Equivariant Student for Unsupervised 3D Human Pose Estimationcited by 10
  16. 2021PAUL: Procrustean Autoencoder for Unsupervised Liftingcited by 9
  17. 2021PR-RRN: Pairwise-Regularized Residual-Recursive Networks for Non-rigid Structure-from-Motioncited by 9
  18. 2017Dense Non-rigid Structure-from-Motion Made Easy - A Spatial-Temporal Smoothness based Solutioncited by 2
  19. 2021GTT-Net: Learned Generalized Trajectory Triangulationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 3; 2026: 1.

8 papers.

  1. 2023Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion Modelingcited by 7
  2. 2022Linear Video Transformer with Feature Fixationcited by 6
  3. 2024Deep Non-rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modelingcited by 2
  4. 2025Non-Rigid Structure-from-Motion via Differential Geometry with Recoverable Conformal Scalecited by 1
  5. 2024Non-rigid Structure-from-Motion: Temporally-smooth Procrustean Alignment and Spatially-variant Deformation Modelingcited by 0
  6. 2025Unsupervised 2D-3D lifting of non-rigid objects using local constraintscited by 0
  7. 2025Enhancing Stress-Strain Predictions with Seq2Seq and Cross-Attention based on Small Punch Testcited by 0
  8. 2026SSR: A Training-Free Approach for Streaming 3D Reconstructioncited by 0