# BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Sounds

arXiv: https://arxiv.org/abs/2503.00389 · HTML: https://mlsanity.com/paper/2503.00389/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 1 Mar 2025.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2303.09119/) (2023)
- [TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation](https://mlsanity.com/paper/2404.16752/) (2024)
- [Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2103.14304/) (2021)
- [xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera](https://mlsanity.com/paper/1907.10045/) (2019)
- [Audio to Body Dynamics](https://mlsanity.com/paper/1712.09382/) (2017)
- [QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation](https://mlsanity.com/paper/2305.11094/) (2023)
- [Human Pose Estimation in Extremely Low-Light Conditions](https://mlsanity.com/paper/2303.15410/) (2023)
- [BatVision: Learning to See 3D Spatial Layout with Two Ears](https://mlsanity.com/paper/1912.07011/) (2019)
- [Diffusion Model is a Good Pose Estimator from 3D RF-Vision](https://mlsanity.com/paper/2403.16198/) (2024)
- [AutoSoccerPose: Automated 3D posture Analysis of Soccer Shot Movements](https://mlsanity.com/paper/2405.12070/) (2024)
- [Unsupervised Learning for Human Sensing Using Radio Signals](https://mlsanity.com/paper/2207.02370/) (2022)
- [Modeling and Driving Human Body Soundfields through Acoustic Primitives](https://mlsanity.com/paper/2407.13083/) (2024)
- [Sound Event Detection and Time-Frequency Segmentation from Weakly Labelled Data](https://mlsanity.com/paper/1804.04715/) (2018)
- [Markerless Motion Capture and Biomechanical Analysis Pipeline](https://mlsanity.com/paper/2303.10654/) (2023)
- [Sound Event Localization based on Sound Intensity Vector Refined By DNN-Based Denoising and Source Separation](https://mlsanity.com/paper/2002.05994/) (2020)
- [Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio](https://mlsanity.com/paper/2311.06285/) (2023)
- [SoundCam: A Dataset for Finding Humans Using Room Acoustics](https://mlsanity.com/paper/2311.03517/) (2023)
- [PoseKernelLifter: Metric Lifting of 3D Human Pose using Sound](https://mlsanity.com/paper/2112.00216/) (2021)
- [Real-Time Human Pose Estimation on a Smart Walker using Convolutional Neural Networks](https://mlsanity.com/paper/2106.14739/) (2021)

## Built on by (most cited first, 1 of 1)

- [Sound-based Multi-Person 3D Pose Estimation](https://mlsanity.com/paper/2609.04902/) (2026)

## Abstract (arXiv)

We propose BGM2Pose, a non-invasive 3D human pose estimation method using arbitrary music (e.g., background music) as active sensing signals. Unlike existing approaches that significantly limit practicality by employing intrusive chirp signals within the audible range, our method utilizes natural music that causes minimal discomfort to humans. Estimating human poses from standard music presents significant challenges. In contrast to sound sources specifically designed for measurement, regular music varies in both volume and pitch. These dynamic changes in signals caused by music are inevitably mixed with alterations in the sound field resulting from human motion, making it hard to extract reliable cues for pose estimation. To address these challenges, BGM2Pose introduces a Contrastive Pose Extraction Module that employs contrastive learning and hard negative sampling to eliminate musical components from the recorded data, isolating the pose information. Additionally, we propose a Frequency-wise Attention Module that enables the model to focus on subtle acoustic variations attributable to human movement by dynamically computing attention across frequency bands. Experiments suggest that our method outperforms the existing methods, demonstrating substantial potential for real-world applications. Project page: https://yutoshibata07.github.io/bgm2pose-project-page/