mlsanityData to 7 Oct 2026

Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation

arXiv 2106.05969 ↗First posted 10 Jun 2021cs.CV cs.AI cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamics-regulated training. At each timestep, a kinematic model is used to provide a target pose using video evidence and simulation state. Then, a prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model. By factoring in the 6DoF pose of objects (e.g., chairs, boxes) in the scene, we demonstrate for the first time, the ability to estimate physically-plausible 3D human-object interactions using a single wearable camera. We evaluate our egocentric pose estimation method in both controlled laboratory settings and real-world scenarios.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 43 of its 68 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

43 of 68 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  6. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  7. 2017End-to-end Recovery of Human Shape and Posecited by 883
  8. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  9. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  10. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  11. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  12. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
31 more
  1. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  2. 2021Character Controllers Using Motion VAEscited by 182
  3. 2017DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networkscited by 173
  4. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  5. 2019Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Imagecited by 131
  6. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 111
  7. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  8. 2019DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Comparecited by 100
  9. 20203D Human Motion Estimation via Motion Compression and Refinementcited by 98
  10. 2020Contact and Human Dynamics from Monocular Videocited by 94
  11. 2017Exploiting temporal information for 3D pose estimationcited by 89
  12. 2018LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Imagescited by 88
  13. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  14. 2019You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactionscited by 82
  15. 2019xR-EgoPose: Egocentric 3D Human Pose from an HMD Cameracited by 78
  16. 2020Hierarchical Kinematic Human Mesh Recoverycited by 71
  17. 2021Neural Monocular 3D Human Motion Capture with Physical Awarenesscited by 70
  18. 2018Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Cameracited by 67
  19. 2020Video Face Manipulation Detection Through Ensemble of CNNscited by 63
  20. 2020Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesiscited by 63
  21. 2020PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Timecited by 50
  22. 2019In the Wild Human Pose Estimation Using Explicit 2D Features and Intermediate 3D Representationscited by 48
  23. 2019Learning 3D Human Shape and Pose from Dense Body Partscited by 48
  24. 2016Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Videocited by 45
  25. 2016Structured Prediction of 3D Human Pose with Deep Neural Networkscited by 43
  26. 2019Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Controlcited by 39
  27. 2020UniCon: Universal Neural Controller For Physics-based Character Motioncited by 39
  28. 2020Optical Non-Line-of-Sight Physics-based 3D Human Pose Estimationcited by 34
  29. 2016EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras (Extended Abstract)cited by 24
  30. 2018SFV: Reinforcement Learning of Physical Skills from Videoscited by 20
  31. 2019Catch & Carry: Reusable Neural Controllers for Vision-Guided Whole-Body Taskscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 16; 2023: 17; 2024: 32; 2025: 21; 2026: 13.

101 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  3. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 198
  4. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  5. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 137
  6. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 112
  7. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
  8. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 109
  9. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  10. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  11. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 101
  12. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 93
38 more
  1. 2023An Outlook into the Future of Egocentric Visioncited by 84
  2. 2022GIMO: Gaze-Informed Human Motion Prediction in Contextcited by 81
  3. 2022Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generationcited by 75
  4. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  5. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  6. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  7. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  8. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 56
  9. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  10. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  11. 2023EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensorscited by 48
  12. 2022LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensorscited by 43
  13. 2022D&D: Learning Human Dynamics from Dynamic Cameracited by 39
  14. 2022Differentiable Dynamics for Articulated 3d Human Motion Reconstructioncited by 38
  15. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 34
  16. 2023PACE: Human and Camera Motion Estimation from in-the-wild Videoscited by 33
  17. 2023Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinementcited by 31
  18. 2022Scene-aware Egocentric 3D Human Pose Estimationcited by 30
  19. 2024Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Cameracited by 30
  20. 2024EventEgo3D: 3D Human Motion Capture from Egocentric Event Streamscited by 29
  21. 2022Embodied Scene-aware Human Pose Estimationcited by 28
  22. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  23. 2023Motion Question Answering via Modular Motion Programscited by 27
  24. 2023QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensorscited by 26
  25. 2024HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Devicecited by 26
  26. 2024PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarioscited by 24
  27. 2025PARC: Physics-based Augmentation with Reinforcement Learning for Character Controllerscited by 24
  28. 20233D Human Pose Perception from Egocentric Stereo Videoscited by 23
  29. 2024LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environmentcited by 20
  30. 2024PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitationcited by 18
  31. 2024MultiPhys: Multi-Person Physics-aware 3D Motion Estimationcited by 18
  32. 2023Decaf: Monocular Deformation Capture for Face and Hand Interactionscited by 17
  33. 2024Real-Time Simulated Avatar from Head-Mounted Sensorscited by 17
  34. 2025Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Inputcited by 17
  35. 2025UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generationcited by 17
  36. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  37. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 16
  38. 2024Contact-aware Human Motion Generation from Textual Descriptionscited by 15

50 most cited of 101 All 101 (JSON)