# From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coaching

arXiv: https://arxiv.org/abs/2603.26938 · HTML: https://mlsanity.com/paper/2603.26938/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 27 Mar 2026.

## Builds on (most cited first, 30 of 33)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models](https://mlsanity.com/paper/2406.09403/) (2024)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [The Revolution of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2402.12451/) (2024)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment](https://mlsanity.com/paper/1904.04346/) (2019)
- [PromptHMR: Promptable Human Mesh Recovery](https://mlsanity.com/paper/2504.06397/) (2025)
- [Counting Out Time: Class Agnostic Video Repetition Counting in the Wild](https://mlsanity.com/paper/2006.15418/) (2020)
- [A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks](https://mlsanity.com/paper/2408.01319/) (2024)
- [Visual In-Context Prompting](https://mlsanity.com/paper/2311.13601/) (2023)
- [Accurate 3D Body Shape Regression using Metric and Semantic Attributes](https://mlsanity.com/paper/2206.07036/) (2022)
- [Learning Clothing and Pose Invariant 3D Shape Representation for Long-Term Person Re-Identification](https://mlsanity.com/paper/2308.10658/) (2023)
- [From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans](https://mlsanity.com/paper/2509.06607/) (2025)
- [Domain Knowledge-Informed Self-Supervised Representations for Workout Form Assessment](https://mlsanity.com/paper/2202.14019/) (2022)
- [Pose Trainer: Correcting Exercise Posture using Pose Estimation](https://mlsanity.com/paper/2006.11718/) (2020)
- [What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction](https://mlsanity.com/paper/2407.08101/) (2024)

## Built on by (most cited first, 2 of 2)

- [FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models](https://mlsanity.com/paper/2608.08736/) (2026)
- [MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching](https://mlsanity.com/paper/2608.26094/) (2026)

## Abstract (arXiv)

We present BioCoach, a biomechanics-grounded vision--language framework for fitness coaching from streaming video. BioCoach fuses visual appearance and 3D skeletal kinematics, through a novel three-stage pipeline: an exercise-specific degree-of-freedom selector that focuses analysis on salient joints; a structured biomechanical context that pairs individualized morphometrics with cycle and constraint analysis; and a vision--biomechanics conditioned feedback module that applies cross-attention to generate precise, actionable text. Using parameter-efficient training that freezes the vision and language backbones, BioCoach yields transparent, personalized reasoning rather than pattern matching. To enable learning and fair evaluation, we augment QEVD-fit-coach with biomechanics-oriented feedback to create QEVD-bio-fit-coach, and we introduce a biomechanics-aware LLM judge metric. BioCoach delivers clear gains on QEVD-bio-fit-coach across lexical and judgment metrics while maintaining temporal triggering; on the original QEVD-fit-coach, it improves text quality and correctness with near-parity timing, demonstrating that explicit kinematics and constraints are key to accurate, phase-aware coaching.