# ELMO: Enhanced Real-time LiDAR Motion Capture through Upsampling

arXiv: https://arxiv.org/abs/2410.06963 · HTML: https://mlsanity.com/paper/2410.06963/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 62 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 30 of 34)

- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose](https://mlsanity.com/paper/1611.07828/) (2016)
- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera](https://mlsanity.com/paper/1705.01583/) (2017)
- [BlazePose: On-device Real-time Body Pose tracking](https://mlsanity.com/paper/2006.10204/) (2020)
- [Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs](https://mlsanity.com/paper/1703.08014/) (2017)
- [Learning to Generate Diverse Dance Motions with Transformer](https://mlsanity.com/paper/2008.08171/) (2020)
- [QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars](https://mlsanity.com/paper/2209.09391/) (2022)
- [C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters](https://mlsanity.com/paper/2309.11351/) (2023)
- [Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time](https://mlsanity.com/paper/1810.04703/) (2018)
- [MoGlow: Probabilistic and controllable motion synthesis using normalising flows](https://mlsanity.com/paper/1905.06598/) (2019)
- [NIKI: Neural Inverse Kinematics with Invertible Neural Networks for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2305.08590/) (2023)
- [LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors](https://mlsanity.com/paper/2205.15410/) (2022)
- [Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control](https://mlsanity.com/paper/1806.08666/) (2018)
- [Neural Categorical Priors for Physics-Based Character Control](https://mlsanity.com/paper/2308.07200/) (2023)
- [Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture](https://mlsanity.com/paper/2203.14065/) (2022)
- [3D Segmentation of Humans in Point Clouds with Synthetic Data](https://mlsanity.com/paper/2212.00786/) (2022)
- [GANimator: Neural Motion Synthesis from a Single Sequence](https://mlsanity.com/paper/2205.02625/) (2022)
- [Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDAR](https://mlsanity.com/paper/2211.16951/) (2022)
- [SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data](https://mlsanity.com/paper/2311.02191/) (2023)
- [LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment](https://mlsanity.com/paper/2402.17171/) (2024)
- [RSMT: Real-time Stylized Motion Transition for Characters](https://mlsanity.com/paper/2306.11970/) (2023)

## Built on by (most cited first, 1 of 1)

- [Point2Pose: A Generative Framework for 3D Human Pose Estimation with Multi-View Point Cloud Dataset](https://mlsanity.com/paper/2512.10321/) (2025)

## Abstract (arXiv)

This paper introduces ELMO, a real-time upsampling motion capture framework designed for a single LiDAR sensor. Modeled as a conditional autoregressive transformer-based upsampling motion generator, ELMO achieves 60 fps motion capture from a 20 fps LiDAR point cloud sequence. The key feature of ELMO is the coupling of the self-attention mechanism with thoughtfully designed embedding modules for motion and point clouds, significantly elevating the motion quality. To facilitate accurate motion capture, we develop a one-time skeleton calibration model capable of predicting user skeleton offsets from a single-frame point cloud. Additionally, we introduce a novel data augmentation technique utilizing a LiDAR simulator, which enhances global root tracking to improve environmental understanding. To demonstrate the effectiveness of our method, we compare ELMO with state-of-the-art methods in both image-based and point cloud-based motion capture. We further conduct an ablation study to validate our design principles. ELMO's fast inference time makes it well-suited for real-time applications, exemplified in our demo video featuring live streaming and interactive gaming scenarios. Furthermore, we contribute a high-quality LiDAR-mocap synchronized dataset comprising 20 different subjects performing a range of motions, which can serve as a valuable resource for future research. The dataset and evaluation code are available at blue urlhttps://movin3d.github.io/ELMOSIGASIA2024/