Can Action be Imitated? Learn to Reconstruct and Transfer Human Dynamics from Videos
Not on the map. Why?
Abstract (arXiv)
Given a video demonstration, can we imitate the action contained in this video? In this paper, we introduce a novel task, dubbed mesh-based action imitation. The goal of this task is to enable an arbitrary target human mesh to perform the same action shown on the video demonstration. To achieve this, a novel Mesh-based Video Action Imitation (M-VAI) method is proposed by us. M-VAI first learns to reconstruct the meshes from the given source image frames, then the initial recovered mesh sequence is fed into mesh2mesh, a mesh sequence smooth module proposed by us, to improve the temporal consistency. Finally, we imitate the actions by transferring the pose from the constructed human body to our target identity mesh. High-quality and detailed human body meshes can be generated by using our M-VAI. Extensive experiments demonstrate the feasibility of our task and the effectiveness of our proposed method.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 25 Jul 2021.
Builds on
32 of 55 references.
References · 32
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
- 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
- 2017End-to-end Recovery of Human Shape and Posecited by 885
- 2015End-to-End Training of Deep Visuomotor Policiescited by 875
- 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 695
- 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 627
- 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 620
- 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 403
- 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 387
- 2018Everybody Dance Nowcited by 315
20 more
- 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
- 2017Pose Guided Person Image Generationcited by 251
- 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
- 2019Multi-Garment Net: Learning to Dress 3D People from Imagescited by 245
- 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
- 2017Single-Shot Multi-Person 3D Pose Estimation From Monocular RGBcited by 195
- 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
- 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
- 2018BodyNet: Volumetric Inference of 3D Human Body Shapescited by 141
- 2020Coherent Reconstruction of Multiple Humans from a Single Imagecited by 136
- 2017VNect: Real-time 3D Human Pose Estimation with a Single RGB Cameracited by 129
- 2019Exploiting temporal context for 3D human pose estimation in the wildcited by 110
- 2019Dense Intrinsic Appearance Flow for Human Pose Transfercited by 104
- 2020Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flowscited by 94
- 2017Towards Accurate Markerless Human Shape and Pose Estimation over Timecited by 92
- 2017Exploiting temporal information for 3D pose estimationcited by 89
- 2018Dense Pose Transfercited by 83
- 2020Neural Pose Transfer by Spatially Adaptive Instance Normalizationcited by 38
- 2017MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Priorcited by 36
- 2020RigNet: Neural Rigging for Articulated Characterscited by 32
Built on by
Citing papers per year: 2022: 1.
1 paper.