ELMO: Enhanced Real-time LiDAR Motion Capture through Upsampling
Not on the map. Why?
Abstract (arXiv)
This paper introduces ELMO, a real-time upsampling motion capture framework designed for a single LiDAR sensor. Modeled as a conditional autoregressive transformer-based upsampling motion generator, ELMO achieves 60 fps motion capture from a 20 fps LiDAR point cloud sequence. The key feature of ELMO is the coupling of the self-attention mechanism with thoughtfully designed embedding modules for motion and point clouds, significantly elevating the motion quality. To facilitate accurate motion capture, we develop a one-time skeleton calibration model capable of predicting user skeleton offsets from a single-frame point cloud. Additionally, we introduce a novel data augmentation technique utilizing a LiDAR simulator, which enhances global root tracking to improve environmental understanding. To demonstrate the effectiveness of our method, we compare ELMO with state-of-the-art methods in both image-based and point cloud-based motion capture. We further conduct an ablation study to validate our design principles. ELMO's fast inference time makes it well-suited for real-time applications, exemplified in our demo video featuring live streaming and interactive gaming scenarios. Furthermore, we contribute a high-quality LiDAR-mocap synchronized dataset comprising 20 different subjects performing a range of motions, which can serve as a valuable resource for future research. The dataset and evaluation code are available at blue urlhttps://movin3d.github.io/ELMOSIGASIA2024/
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 62 references that are in the map. First posted on arXiv 9 Oct 2024.
Builds on
34 of 62 references.
References · 34
- 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
- 2022Human Motion Diffusion Modelcited by 1,080
- 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
- 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 695
- 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 620
- 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
- 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 350
- 2022EDGE: Editable Dance Generation From Musiccited by 287
- 2016Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Posecited by 221
- 2021Character Controllers Using Motion VAEscited by 184
- 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 142
- 2017VNect: Real-time 3D Human Pose Estimation with a Single RGB Cameracited by 129
22 more
- 2020BlazePose: On-device Real-time Body Pose trackingcited by 113
- 2017Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUscited by 109
- 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
- 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 77
- 2023C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characterscited by 55
- 2018Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Timecited by 47
- 2019MoGlow: Probabilistic and controllable motion synthesis using normalising flowscited by 47
- 2023NIKI: Neural Inverse Kinematics with Invertible Neural Networks for 3D Human Pose and Shape Estimationcited by 45
- 2022LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensorscited by 44
- 2018Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Controlcited by 39
- 2023Neural Categorical Priors for Physics-Based Character Controlcited by 38
- 2022Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capturecited by 31
- 20223D Segmentation of Humans in Point Clouds with Synthetic Datacited by 24
- 2022GANimator: Neural Motion Synthesis from a Single Sequencecited by 23
- 2022Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDARcited by 23
- 2023SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Datacited by 21
- 2024LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environmentcited by 20
- 2023RSMT: Real-time Stylized Motion Transition for Characterscited by 19
- 2023MuscleVAE: Model-Based Controllers of Muscle-Actuated Characterscited by 13
- 2023ACE: Adversarial Correspondence Embedding for Cross Morphology Motion Retargeting from Human to Nonhuman Characterscited by 12
- 2023MOCHA: Real-Time Motion Characterization via Context Matchingcited by 8
- 2023MOVIN: Real-time Motion Capture using a Single LiDARcited by 6
Built on by
Citing papers per year: 2025: 1.
1 paper.