# Self-Supervised Learning of Gait-Based Biomarkers

arXiv: https://arxiv.org/abs/2307.16321 · HTML: https://mlsanity.com/paper/2307.16321/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 30 Jul 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Equinox: neural networks in JAX via callable PyTrees and filtered transformations](https://mlsanity.com/paper/2111.00254/) (2021)
- [MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators](https://mlsanity.com/paper/2306.10900/) (2023)
- [The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies](https://mlsanity.com/paper/2007.15911/) (2020)
- [MoVi: A Large Multipurpose Motion and Video Dataset](https://mlsanity.com/paper/2003.01888/) (2020)
- [Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats](https://mlsanity.com/paper/2212.14474/) (2022)
- [Improved Trajectory Reconstruction for Markerless Pose Estimation](https://mlsanity.com/paper/2303.02413/) (2023)
- [GaitForeMer: Self-Supervised Pre-Training of Transformers via Human Motion Forecasting for Few-Shot Gait Impairment Severity Estimation](https://mlsanity.com/paper/2207.00106/) (2022)
- [PosePipe: Open-Source Human Pose Estimation Pipeline for Clinical Research](https://mlsanity.com/paper/2203.08792/) (2022)
- [Markerless Motion Capture and Biomechanical Analysis Pipeline](https://mlsanity.com/paper/2303.10654/) (2023)
- [Transforming Gait: Video-Based Spatiotemporal Gait Analysis](https://mlsanity.com/paper/2203.09371/) (2022)
- [Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models](https://mlsanity.com/paper/2210.13432/) (2022)

## Built on by (most cited first, 5 of 5)

- [Differentiable Biomechanics Unlocks Opportunities for Markerless Motion Capture](https://mlsanity.com/paper/2402.17192/) (2024)
- [Fusing uncalibrated IMUs and handheld smartphone video to reconstruct knee kinematics](https://mlsanity.com/paper/2405.17368/) (2024)
- [Advancing Monocular Video-Based Gait Analysis Using Motion Imitation with Physics-Based Simulation](https://mlsanity.com/paper/2402.12676/) (2024)
- [A Causal Framework for Precision Rehabilitation](https://mlsanity.com/paper/2411.03919/) (2024)
- [A Gait Foundation Model Predicts Multi-System Health Phenotypes from 3D Skeletal Motion](https://mlsanity.com/paper/2603.25283/) (2026)

## Abstract (arXiv)

Markerless motion capture (MMC) is revolutionizing gait analysis in clinical settings by making it more accessible, raising the question of how to extract the most clinically meaningful information from gait data. In multiple fields ranging from image processing to natural language processing, self-supervised learning (SSL) from large amounts of unannotated data produces very effective representations for downstream tasks. However, there has only been limited use of SSL to learn effective representations of gait and movement, and it has not been applied to gait analysis with MMC. One SSL objective that has not been applied to gait is contrastive learning, which finds representations that place similar samples closer together in the learned space. If the learned similarity metric captures clinically meaningful differences, this could produce a useful representation for many downstream clinical tasks. Contrastive learning can also be combined with causal masking to predict future timesteps, which is an appealing SSL objective given the dynamical nature of gait. We applied these techniques to gait analyses performed with MMC in a rehabilitation hospital from a diverse clinical population. We find that contrastive learning on unannotated gait data learns a representation that captures clinically meaningful information. We probe this learned representation using the framework of biomarkers and show it holds promise as both a diagnostic and response biomarker, by showing it can accurately classify diagnosis from gait and is responsive to inpatient therapy, respectively. We ultimately hope these learned representations will enable predictive and prognostic gait-based biomarkers that can facilitate precision rehabilitation through greater use of MMC to quantify movement in rehabilitation.