# SOMA: Solving Optical Marker-Based MoCap Automatically

arXiv: https://arxiv.org/abs/2110.04431 · HTML: https://mlsanity.com/paper/2110.04431/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 62 references that are in the map.
- First posted on arXiv 9 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [MoVi: A Large Multipurpose Motion and Video Dataset](https://mlsanity.com/paper/2003.01888/) (2020)
- [Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs](https://mlsanity.com/paper/1606.07253/) (2016)
- [Auto-labelling of Markers in Optical Motion Capture by Permutation Learning](https://mlsanity.com/paper/1907.13580/) (2019)

## Built on by (most cited first, 30 of 31)

- [SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos](https://mlsanity.com/paper/2112.13715/) (2021)
- [FLAG: Flow-based 3D Avatar Generation from Sparse Observations](https://mlsanity.com/paper/2203.05789/) (2022)
- [Digital Life Project: Autonomous 3D Characters with Social Intelligence](https://mlsanity.com/paper/2312.04547/) (2023)
- [AnimateAnyMesh: A Feed-Forward 4D Foundation Model for Text-Driven Universal Mesh Animation](https://mlsanity.com/paper/2506.09982/) (2025)
- [HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations](https://mlsanity.com/paper/2403.03561/) (2024)
- [BEDLAM2.0: Synthetic Humans and Cameras in Motion](https://mlsanity.com/paper/2511.14394/) (2025)
- [SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data](https://mlsanity.com/paper/2311.02191/) (2023)
- [HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observations](https://mlsanity.com/paper/2308.11261/) (2023)
- [Ultra Inertial Poser: Scalable Motion Capture and Tracking from Sparse Inertial Sensors and Ultra-Wideband Ranging](https://mlsanity.com/paper/2404.19541/) (2024)
- [PRIMAL: Physically Reactive and Interactive Motor Model for Avatar Learning](https://mlsanity.com/paper/2503.17544/) (2025)
- [SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation](https://mlsanity.com/paper/2411.19921/) (2024)
- [From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models](https://mlsanity.com/paper/2504.05265/) (2025)
- [M4Human: A Large-Scale Multimodal mmWave Radar Benchmark for Human Mesh Reconstruction](https://mlsanity.com/paper/2512.12378/) (2025)
- [RoMo: A Robust Solver for Full-body Unlabeled Optical Motion Capture](https://mlsanity.com/paper/2410.02788/) (2024)
- [A Locality-based Neural Solver for Optical Motion Capture](https://mlsanity.com/paper/2309.00428/) (2023)
- [DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimization](https://mlsanity.com/paper/2406.14567/) (2024)
- [NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data](https://mlsanity.com/paper/2603.18496/) (2026)
- [Towards Unstructured Unlabeled Optical Mocap: A Video Helps!](https://mlsanity.com/paper/2407.06114/) (2024)
- [RoMo: A Large-Scale, Richly Organized Dataset and Semantic Taxonomy for Human Motion Generation](https://mlsanity.com/paper/2605.26241/) (2026)
- [MMCM: Multimodality-aware Metric using Clustering-based Modes for Probabilistic Human Motion Prediction](https://mlsanity.com/paper/2511.15179/) (2025)
- [A Light-Weight Contrastive Approach for Aligning Human Pose Sequences](https://mlsanity.com/paper/2303.04244/) (2023)
- [Noise-in, Bias-out: Balanced and Real-time MoCap Solving](https://mlsanity.com/paper/2309.14330/) (2023)
- [BiomechGPT: Extending Motion-Language Models to Clinical Motion Understanding](https://mlsanity.com/paper/2505.18465/) (2025)
- [L-SR1: Learned Symmetric-Rank-One Preconditioning](https://mlsanity.com/paper/2508.12270/) (2025)
- [End-to-End Motion Capture from Rigid Body Markers with Geodesic Loss](https://mlsanity.com/paper/2511.16418/) (2025)
- [Garment Inertial Denoiser (GID): Endowing Accurate Motion Capture via Loose IMU Denoiser](https://mlsanity.com/paper/2601.01360/) (2026)
- [DexterCap: An Affordable and Automated System for Capturing Dexterous Hand-Object Manipulation](https://mlsanity.com/paper/2601.05844/) (2026)
- [Human Pose Estimation in Trampoline Gymnastics: How to Improve Performance on Extreme Poses](https://mlsanity.com/paper/2604.01322/) (2026)
- [Map-Mono-Ego: Map-Grounded Global Human Pose Estimation from Monocular Egocentric Video](https://mlsanity.com/paper/2605.20889/) (2026)
- [DirtyMoCap: Robust Motion Capture from Unconstrained Markers](https://mlsanity.com/paper/2609.19927/) (2026)

## Abstract (arXiv)

Marker-based optical motion capture (mocap) is the "gold standard" method for acquiring accurate 3D human motion in computer vision, medicine, and graphics. The raw output of these systems are noisy and incomplete 3D points or short tracklets of points. To be useful, one must associate these points with corresponding markers on the captured subject; i.e. "labelling". Given these labels, one can then "solve" for the 3D skeleton or body surface mesh. Commercial auto-labeling tools require a specific calibration procedure at capture time, which is not possible for archival data. Here we train a novel neural network called SOMA, which takes raw mocap point clouds with varying numbers of points, labels them at scale without any calibration data, independent of the capture technology, and requiring only minimal human intervention. Our key insight is that, while labeling point clouds is highly ambiguous, the 3D body provides strong constraints on the solution that can be exploited by a learning-based method. To enable learning, we generate massive training sets of simulated noisy and ground truth mocap markers animated by 3D bodies from AMASS. SOMA exploits an architecture with stacked self-attention elements to learn the spatial structure of the 3D body and an optimal transport layer to constrain the assignment (labeling) problem while rejecting outliers. We extensively evaluate SOMA both quantitatively and qualitatively. SOMA is more accurate and robust than existing state of the art research methods and can be applied where commercial systems cannot. We automatically label over 8 hours of archival mocap data across 4 different datasets captured using various technologies and output SMPL-X body models. The model and data is released for research purposes at https://soma.is.tue.mpg.de/.