Neural Marionette: Unsupervised Learning of Motion Skeleton and Latent Dynamics from Volumetric Video
Not on the map. Why?
Abstract (arXiv)
We present Neural Marionette, an unsupervised approach that discovers the skeletal structure from a dynamic sequence and learns to generate diverse motions that are consistent with the observed motion dynamics. Given a video stream of point cloud observation of an articulated body under arbitrary motion, our approach discovers the unknown low-dimensional skeletal relationship that can effectively represent the movement. Then the discovered structure is utilized to encode the motion priors of dynamic sequences in a latent structure, which can be decoded to the relative joint rotations to represent the full skeletal motion. Our approach works without any prior knowledge of the underlying motion or skeletal structure, and we demonstrate that the discovered structure is even comparable to the hand-labeled ground truth skeleton in representing a 4D sequence of motion. The skeletal structure embeds the general semantics of possible motion space that can generate motions for diverse scenarios. We verify that the learned motion prior is generalizable to the multi-modal sequence generation, interpolation of two poses, and motion retargeting to a different skeletal structure.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 17 Feb 2022.
Builds on
37 of 49 references.
References · 37
- 2013Auto-Encoding Variational Bayescited by 3,980
- 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
- 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
- 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
- 2015Unsupervised Learning of Video Representations using LSTMscited by 434
- 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
- 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
- 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 339
- 20163D Menagerie: Modeling the 3D shape and pose of animalscited by 310
- 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 302
- 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
- 2019Learning Trajectory Dependencies for Human Motion Predictioncited by 245
25 more
- 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
- 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
- 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
- 2019Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboidscited by 123
- 20214DComplete: Non-Rigid Motion Estimation Beyond the Observable Surfacecited by 123
- 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 113
- 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
- 2021Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networkscited by 101
- 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
- 2021NPMs: Neural Parametric Models for 3D Deformable Shapescited by 89
- 2019Variational Graph Recurrent Neural Networkscited by 86
- 2018Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoningcited by 79
- 2020Causal Discovery in Physical Systems from Videoscited by 77
- 2020Improving Generative Imagination in Object-Centric World Modelscited by 70
- 2020Neural Deformation Graphs for Globally-consistent Non-rigid Reconstructioncited by 66
- 2020CaSPR: Learning Canonical Spatiotemporal Point Cloud Representationscited by 51
- 2019Predicting Animation Skeletons for 3D Articulated Models via Volumetric Netscited by 39
- 2020Point2Skeleton: Learning Skeletal Representations from Point Cloudscited by 37
- 2020TransMoMo: Invariance-Driven Unsupervised Video Motion Retargetingcited by 36
- 2019Human Motion Prediction via Learning Local Structure Representations and Temporal Dependenciescited by 34
- 2021Contrastive Representation Learning for Hand Shape Estimationcited by 34
- 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
- 2021Unsupervised Human Pose Estimation through Transforming Shape Templatescited by 26
- 2019Unsupervised Discovery of Parts, Structure, and Dynamicscited by 20
- 2020Learning a Decentralized Multi-arm Motion Plannercited by 17
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.