# Sound-based Multi-Person 3D Pose Estimation

arXiv: https://arxiv.org/abs/2609.04902 · HTML: https://mlsanity.com/paper/2609.04902/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [Person-in-WiFi: Fine-grained Person Perception using WiFi](https://mlsanity.com/paper/1904.00276/) (2019)
- [PSVT: End-to-End Multi-person 3D Pose and Shape Estimation with Progressive Video Transformers](https://mlsanity.com/paper/2303.09187/) (2023)
- [LSTM Pose Machines](https://mlsanity.com/paper/1712.06316/) (2017)
- [Human Pose Estimation in Extremely Low-Light Conditions](https://mlsanity.com/paper/2303.15410/) (2023)
- [Group Pose: A Simple Baseline for End-to-End Multi-person Pose Estimation](https://mlsanity.com/paper/2308.07313/) (2023)
- [STFNets: Learning Sensing Signals from the Time-Frequency Perspective with Short-Time Fourier Neural Networks](https://mlsanity.com/paper/1902.07849/) (2019)
- [Snipper: A Spatiotemporal Transformer for Simultaneous Multi-Person 3D Pose Estimation Tracking and Forecasting on a Video Snippet](https://mlsanity.com/paper/2207.04320/) (2022)
- [hear-your-action: human action recognition by ultrasound active sensing](https://mlsanity.com/paper/2309.08087/) (2023)
- [Acoustic-based 3D Human Pose Estimation Robust to Human Position](https://mlsanity.com/paper/2411.07165/) (2024)
- [RAPTR: Radar-based 3D Pose Estimation using Transformer](https://mlsanity.com/paper/2511.08387/) (2025)
- [Sonicmesh: Enhancing 3D Human Mesh Reconstruction in Vision-Impaired Environments With Acoustic Signals](https://mlsanity.com/paper/2412.11325/) (2024)
- [BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Sounds](https://mlsanity.com/paper/2503.00389/) (2025)

## Abstract (arXiv)

Can we recover the 3D poses of multiple people using only sound? This paper presents the first attempt to estimate multi-person 3D poses solely from acoustic signals. Estimating the poses of multiple individuals using acoustic signals is inherently challenging due to the superposition of motion-dependent signal variations. Unlike single-person scenarios, the presence of multiple subjects leads to overlapping acoustic signatures, making it difficult to attribute specific signal changes to an individual's pose. Furthermore, the complexity is compounded by inter-person reflections, which introduce intricate propagation delays that obscure the temporal motion-acoustic relationship. To address these issues, we propose SoundMHPE (Sound-based Multi-person Human Pose Estimator), a novel encoder-decoder framework consisting of two key components. First, the Acoustic Multi-scale Encoder captures diverse temporal and fine-grained frequency features to isolate subtle acoustic signatures from complex, overlapping signals. Second, the Temporal Pose Decoder employs an attention mechanism to disentangle multi-person information across successive frames. By jointly accounting for temporal dynamics and inter-person dependencies, this component precisely reconstructs frame-wise individual poses. To validate our approach, we constructed the 6-hour Acoustic Multi-person Pose (AMP) dataset consisting of 432K synchronized frames of multi-person pose and acoustic data, and demonstrated that our SoundMHPE outperforms baseline models. Project page: https://oumi03.github.io/sound-mhpe/