# DEMOS: Dynamic Environment Motion Synthesis in 3D Scenes via Local Spherical-BEV Perception

arXiv: https://arxiv.org/abs/2403.01740 · HTML: https://mlsanity.com/paper/2403.01740/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 55 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes](https://mlsanity.com/paper/2012.05522/) (2020)
- [Generating 3D People in Scenes without People](https://mlsanity.com/paper/1912.02923/) (2019)
- [Efficient Learning on Point Clouds with Basis Point Sets](https://mlsanity.com/paper/1908.09186/) (2019)
- [Learning Motion Priors for 4D Human Body Capture in 3D Scenes](https://mlsanity.com/paper/2108.10399/) (2021)
- [Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis](https://mlsanity.com/paper/2205.13001/) (2022)
- [Compositional Human-Scene Interaction Synthesis with Semantic Control](https://mlsanity.com/paper/2207.12824/) (2022)
- [Scene-aware Generative Network for Human Motion Synthesis](https://mlsanity.com/paper/2105.14804/) (2021)
- [Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds](https://mlsanity.com/paper/1909.09287/) (2019)
- [MIME: Human-Aware 3D Scene Generation](https://mlsanity.com/paper/2212.04360/) (2022)
- [Kernel Transformer Networks for Compact Spherical Convolution](https://mlsanity.com/paper/1812.03115/) (2018)
- [Learning Semantic Segmentation of Large-Scale Point Clouds with Random Sampling](https://mlsanity.com/paper/2107.02389/) (2021)
- [LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors](https://mlsanity.com/paper/2205.15410/) (2022)
- [Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control](https://mlsanity.com/paper/1806.08666/) (2018)

## Abstract (arXiv)

Motion synthesis in real-world 3D scenes has recently attracted much attention. However, the static environment assumption made by most current methods usually cannot be satisfied especially for real-time motion synthesis in scanned point cloud scenes, if multiple dynamic objects exist, e.g., moving persons or vehicles. To handle this problem, we propose the first Dynamic Environment MOtion Synthesis framework (DEMOS) to predict future motion instantly according to the current scene, and use it to dynamically update the latent motion for final motion synthesis. Concretely, we propose a Spherical-BEV perception method to extract local scene features that are specifically designed for instant scene-aware motion prediction. Then, we design a time-variant motion blending to fuse the new predicted motions into the latent motion, and the final motion is derived from the updated latent motions, benefitting both from motion-prior and iterative methods. We unify the data format of two prevailing datasets, PROX and GTA-IM, and take them for motion synthesis evaluation in 3D scenes. We also assess the effectiveness of the proposed method in dynamic environments from GTA-IM and Semantic3D to check the responsiveness. The results show our method outperforms previous works significantly and has great performance in handling dynamic environments.