# A Locality-based Neural Solver for Optical Motion Capture

arXiv: https://arxiv.org/abs/2309.00428 · HTML: https://mlsanity.com/paper/2309.00428/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 48 references that are in the map.
- First posted on arXiv 1 Sep 2023.

## Builds on (most cited first, 14 of 14)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [Skeleton-Aware Networks for Deep Motion Retargeting](https://mlsanity.com/paper/2005.05732/) (2020)
- [SOMA: Solving Optical Marker-Based MoCap Automatically](https://mlsanity.com/paper/2110.04431/) (2021)
- [Part Aware Contrastive Learning for Self-Supervised Action Recognition](https://mlsanity.com/paper/2305.00666/) (2023)
- [Deep unsupervised 3D human body reconstruction from a sparse set of landmarks](https://mlsanity.com/paper/2106.12282/) (2021)

## Built on by (most cited first, 5 of 5)

- [StableMotion: Training Motion Cleanup Models with Unpaired Corrupted Data](https://mlsanity.com/paper/2505.03154/) (2025)
- [Towards Unstructured Unlabeled Optical Mocap: A Video Helps!](https://mlsanity.com/paper/2407.06114/) (2024)
- [OpenMoCap: Rethinking Optical Motion Capture under Real-world Occlusion](https://mlsanity.com/paper/2508.12610/) (2025)
- [Evaluating the Evaluators: Towards Human-aligned Metrics for Missing Markers Reconstruction](https://mlsanity.com/paper/2410.14334/) (2024)
- [DexterCap: An Affordable and Automated System for Capturing Dexterous Hand-Object Manipulation](https://mlsanity.com/paper/2601.05844/) (2026)

## Abstract (arXiv)

We present a novel locality-based learning method for cleaning and solving optical motion capture data. Given noisy marker data, we propose a new heterogeneous graph neural network which treats markers and joints as different types of nodes, and uses graph convolution operations to extract the local features of markers and joints and transform them to clean motions. To deal with anomaly markers (e.g. occluded or with big tracking errors), the key insight is that a marker's motion shows strong correlations with the motions of its immediate neighboring markers but less so with other markers, a.k.a. locality, which enables us to efficiently fill missing markers (e.g. due to occlusion). Additionally, we also identify marker outliers due to tracking errors by investigating their acceleration profiles. Finally, we propose a training regime based on representation learning and data augmentation, by training the model on data with masking. The masking schemes aim to mimic the occluded and noisy markers often observed in the real data. Finally, we show that our method achieves high accuracy on multiple metrics across various datasets. Extensive comparison shows our method outperforms state-of-the-art methods in terms of prediction accuracy of occluded marker position error by approximately 20%, which leads to a further error reduction on the reconstructed joint rotations and positions by 30%. The code and data for this paper are available at https://github.com/non-void/LocalMoCap.