# Muscles in Time: Learning to Understand Human Motion by Simulating Muscle Activations

arXiv: https://arxiv.org/abs/2411.00128 · HTML: https://mlsanity.com/paper/2411.00128/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 82 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [Efficient Parameter-free Clustering Using First Neighbor Relations](https://mlsanity.com/paper/1902.11266/) (2019)
- [SeqSleepNet: End-to-End Hierarchical Recurrent Neural Network for Sequence-to-Sequence Automatic Sleep Staging](https://mlsanity.com/paper/1809.10932/) (2018)
- [Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2208.10741/) (2022)
- [Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition](https://mlsanity.com/paper/2007.14690/) (2020)
- [MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control](https://mlsanity.com/paper/2205.13600/) (2022)
- [Spatial Temporal Transformer Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2012.06399/) (2020)
- [Imputer: Sequence Modelling via Imputation and Dynamic Programming](https://mlsanity.com/paper/2002.08926/) (2020)
- [Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model](https://mlsanity.com/paper/1910.10697/) (2019)
- [Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models](https://mlsanity.com/paper/1701.03185/) (2017)
- [Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation](https://mlsanity.com/paper/2210.12649/) (2022)
- [SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training](https://mlsanity.com/paper/2307.08476/) (2023)
- [Deep Reinforcement Learning For Sequence to Sequence Models](https://mlsanity.com/paper/1805.09461/) (2018)
- [Hypergraph Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2211.09590/) (2022)
- [Guiding attention in Sequence-to-sequence models for Dialogue Act prediction](https://mlsanity.com/paper/2002.08801/) (2020)
- [Neural Machine Translation and Sequence-to-sequence Models: A Tutorial](https://mlsanity.com/paper/1703.01619/) (2017)

## Built on by (most cited first, 3 of 3)

- [MotionWavelet: Human Motion Prediction via Wavelet Manifold Learning](https://mlsanity.com/paper/2411.16964/) (2024)
- [Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learning](https://mlsanity.com/paper/2601.21904/) (2026)
- [SOMA: From Surface Observations to Muscle Anatomy](https://mlsanity.com/paper/2606.09246/) (2026)

## Abstract (arXiv)

Exploring the intricate dynamics between muscular and skeletal structures is pivotal for understanding human motion. This domain presents substantial challenges, primarily attributed to the intensive resources required for acquiring ground truth muscle activation data, resulting in a scarcity of datasets. In this work, we address this issue by establishing Muscles in Time (MinT), a large-scale synthetic muscle activation dataset. For the creation of MinT, we enriched existing motion capture datasets by incorporating muscle activation simulations derived from biomechanical human body models using the OpenSim platform, a common approach in biomechanics and human motion research. Starting from simple pose sequences, our pipeline enables us to extract detailed information about the timing of muscle activations within the human musculoskeletal system. Muscles in Time contains over nine hours of simulation data covering 227 subjects and 402 simulated muscle strands. We demonstrate the utility of this dataset by presenting results on neural network-based muscle activation estimation from human pose sequences with two different sequence-to-sequence architectures. Data and code are provided under https://simplexsigil.github.io/mint.