mlsanityData to 7 Oct 2026

Utilizing Task-Generic Motion Prior to Recover Full-Body Motion from Very Sparse Signals

arXiv 2308.15839 ↗First posted 30 Aug 2023cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

The most popular type of devices used to track a user's posture in a virtual reality experience consists of a head-mounted display and two controllers held in both hands. However, due to the limited number of tracking sensors (three in total), faithfully recovering the user in full-body is challenging, limiting the potential for interactions among simulated user avatars within the virtual world. Therefore, recent studies have attempted to reconstruct full-body poses using neural networks that utilize previously learned human poses or accept a series of past poses over a short period. In this paper, we propose a method that utilizes information from a neural motion prior to improve the accuracy of reconstructed user's motions. Our approach aims to reconstruct user's full-body poses by predicting the latent representation of the user's overall motion from limited input signals and integrating this information with tracking sensor inputs. This is based on the premise that the ultimate goal of pose reconstruction is to reconstruct the motion, which is a series of poses. Our results show that this integration enables more accurate reconstruction of the user's full-body motion, particularly enhancing the robustness of lower body motion reconstruction from impoverished signals. Web: https://https://mjsh34.github.io/mp-sspe/

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 44 of its 73 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

44 of 73 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2016Density estimation using Real NVPcited by 2,029
  5. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  6. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  7. 2022Human Motion Diffusion Modelcited by 1,062
  8. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  9. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  10. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  11. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  12. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
32 more
  1. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  2. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  3. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  4. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  5. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  6. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  7. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  8. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  9. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  10. 2021Character Controllers Using Motion VAEscited by 182
  11. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 162
  12. 2019Learnable Triangulation of Human Posecited by 160
  13. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 152
  14. 2022Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensorscited by 144
  15. 2017VNect: Real-time 3D Human Pose Estimation with a Single RGB Cameracited by 129
  16. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 111
  17. 2017Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUscited by 108
  18. 2022AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensingcited by 107
  19. 20203D Human Motion Estimation via Motion Compression and Refinementcited by 98
  20. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  21. 2019You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactionscited by 82
  22. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  23. 2022NeMF: Neural Motion Fields for Kinematic Animationcited by 66
  24. 2022FLAG: Flow-based 3D Avatar Generation from Sparse Observationscited by 54
  25. 2021THUNDR: Transformer-based 3D HUmaN Reconstruction with Markerscited by 50
  26. 2016Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Videocited by 45
  27. 2021LoBSTr: Real-time Lower-body Pose Prediction from Sparse Upper-body Tracking Signalscited by 45
  28. 2021Task-Generic Hierarchical Human Motion Prior using VAEscited by 33
  29. 2020MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistencycited by 31
  30. 2021Gravity-Aware Monocular 3D Human-Object Reconstructioncited by 30
  31. 2022Neural3Points: Learning to Generate Physically Realistic Full-body Motion for Virtual Reality Userscited by 20
  32. 2022Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer-Grade VR Devicescited by 13

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimizationcited by 3
  2. 2024Self-Avatar Animation in Virtual Reality: Impact of Motion Signals Artifacts on the Full-Body Pose Reconstructioncited by 2