# From Pixels to Newtons: Predicting In Vivo Joint Contact Forces from Monocular Video

arXiv: https://arxiv.org/abs/2606.06631 · HTML: https://mlsanity.com/paper/2606.06631/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 80 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 18 of 18)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks](https://mlsanity.com/paper/2203.09168/) (2022)
- [Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation](https://mlsanity.com/paper/2401.08559/) (2024)
- [Evaluating and Calibrating Uncertainty Prediction in Regression Tasks](https://mlsanity.com/paper/1905.11659/) (2019)
- [PromptHMR: Promptable Human Mesh Recovery](https://mlsanity.com/paper/2504.06397/) (2025)
- [SAM 3D Body: Robust Full-Body Human Mesh Recovery](https://mlsanity.com/paper/2602.15989/) (2026)
- [Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2407.07532/) (2024)
- [OpenCap Monocular: 3D Human Kinematics and Musculoskeletal Dynamics from a Single Smartphone Video](https://mlsanity.com/paper/2603.24733/) (2026)

## Abstract (arXiv)

Joint contact forces govern implant longevity, cartilage health, and rehabilitation outcomes, shaping who develops osteoarthritis, who recovers well from joint replacement, and who benefits from biomechanical interventions. Yet they remain measurable only invasively, in a few dozen patients with instrumented implants. I present a physics-free pipeline to predict instantaneous 3D hip and knee contact forces from an uncalibrated monocular video: no markers, force plates, electromyography, subject-specific imaging, or musculoskeletal model. Parametric body meshes are recovered per frame, encoded as kinematic features, and decoded into forces by a transformer whose pose stream is adaptively modulated at every layer by body shape, joint, side, activity text, and self-supervised video tokens (V-JEPA 2), unifying hip and knee in a single model. Under leave-one-subject-out cross-validation across 26 patients and 25 activity categories from the in vivo OrthoLoad database, the pipeline matches the accuracy of subject-specific musculoskeletal simulations (0.32 ± 0.08 BW RMSE for hip; 0.23 ± 0.03 BW for knee) and resolves peak force changes smaller than those reported for gait retraining and osteoarthritis progression. Applied zero-shot to an independent instrumented cohort, it rivals or outperforms prior published methods. Even without curated activity labels, video features alone preserve accuracy and enable end-to-end inference on raw footage. Driven by the predictor, a generative motion prior produces biomechanically plausible variants with reduced peak loading, rediscovering strategies from the predictive simulation literature. This pipeline establishes uncalibrated monocular video as a viable modality for estimating joint loading, opening a path toward retrospective analysis of archived clinical recordings, primary-care screening, and at-home rehabilitation tracking.