mlsanityData to 7 Oct 2026

PoseKernelLifter: Metric Lifting of 3D Human Pose using Sound

arXiv 2112.00216 ↗First posted 1 Dec 2021cs.CV cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Reconstructing the 3D pose of a person in metric scale from a single view image is a geometrically ill-posed problem. For example, we can not measure the exact distance of a person to the camera from a single view image without additional scene assumptions (e.g., known height). Existing learning based approaches circumvent this issue by reconstructing the 3D pose up to scale. However, there are many applications such as virtual telepresence, robotics, and augmented reality that require metric scale reconstruction. In this paper, we show that audio signals recorded along with an image, provide complementary information to reconstruct the metric 3D pose of the person. The key insight is that as the audio signals traverse across the 3D space, their interactions with the body provide metric information about the body's pose. Based on this insight, we introduce a time-invariant transfer function called pose kernel -- the impulse response of audio signals induced by the body pose. The main properties of the pose kernel are that (1) its envelope highly correlates with 3D pose, (2) the time response corresponds to arrival time, indicating the metric distance to the microphone, and (3) it is invariant to changes in the scene geometry configurations. Therefore, it is readily generalizable to unseen scenes. We design a multi-stage 3D CNN that fuses audio and visual signals and learns to reconstruct 3D pose in a metric scale. We show that our multi-modal method produces accurate metric reconstruction in real world scenes, which is not possible with state-of-the-art lifting approaches including parametric mesh regression and depth regression.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 77 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

43 of 77 references.

References · 43

  1. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  2. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  3. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  4. 2017Learning from Synthetic Humanscited by 358
  5. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  6. 2016Convolutional Pose Machinescited by 325
  7. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 304
  8. 2017Integral Human Pose Regressioncited by 255
  9. 2016Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Posecited by 220
  10. 2019Semantic Graph Convolutional Networks for 3D Human Pose Regressioncited by 204
  11. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  12. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
31 more
  1. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  2. 2019Learnable Triangulation of Human Posecited by 160
  3. 2020Epipolar Transformerscited by 117
  4. 2019Co-Separating Sounds of Visual Objectscited by 110
  5. 2019Exploiting temporal context for 3D human pose estimation in the wildcited by 110
  6. 2019Self-Supervised Learning of 3D Human Pose using Multi-view Geometrycited by 98
  7. 2019Unsupervised 3D Pose Estimation with Geometric Self-Supervisioncited by 95
  8. 2021Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimationcited by 94
  9. 2021Graph Stacked Hourglass Networks for 3D Human Pose Estimationcited by 94
  10. 2021PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimationcited by 88
  11. 2019RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimationcited by 85
  12. 2017Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Imagecited by 73
  13. 2018Unsupervised Geometry-Aware Representation for 3D Human Pose Estimationcited by 73
  14. 2018Learning Monocular 3D Human Pose Estimation from Multi-view Imagescited by 70
  15. 2020Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wildcited by 66
  16. 2020CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wildcited by 65
  17. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  18. 20203D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Trainingcited by 54
  19. 2019In the Wild Human Pose Estimation Using Explicit 2D Features and Intermediate 3D Representationscited by 48
  20. 2015Deep Multimodal Learning for Audio-Visual Speech Recognitioncited by 37
  21. 2020Predicting Camera Viewpoint Improves Cross-dataset Generalization for 3D Human Pose Estimationcited by 36
  22. 2019BatVision: Learning to See 3D Spatial Layout with Two Earscited by 26
  23. 2018Unsupervised Adversarial Learning of 3D Human Pose from 2D Joint Locationscited by 18
  24. 2018Multi-Modal Geometric Learning for Grasping and Manipulationcited by 15
  25. 2019PoseLifter: Absolute 3D human pose lifting network from a single noisy 2D human posecited by 10
  26. 2020Metric-Scale Truncation-Robust Heatmaps for 3D Human Pose Estimationcited by 8
  27. 2020LiftFormer: 3D Human Pose Estimation using attention modelscited by 7
  28. 2021AuraSense: Robot Collision Avoidance by Full Surface Proximity Detectioncited by 4
  29. 2018Skeleton Transformer Networks: 3D Human Pose and Skinned Mesh from Single RGB Imagecited by 3
  30. 2021Echo-Reconstruction: Audio-Augmented 3D Scene Reconstructioncited by 2
  31. 2016Multi-modal Tracking for Object based SLAMcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2023SoundCam: A Dataset for Finding Humans Using Room Acousticscited by 5
  2. 2024Acoustic-based 3D Human Pose Estimation Robust to Human Positioncited by 2
  3. 2024CLHOP: Combined Audio-Video Learning for Horse 3D Pose and Shape Estimationcited by 1
  4. 2025BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Soundscited by 1