Muscles in Action
Not on the map. Why?
Abstract (arXiv)
Human motion is created by, and constrained by, our muscles. We take a first step at building computer vision methods that represent the internal muscle activity that causes motion. We present a new dataset, Muscles in Action (MIA), to learn to incorporate muscle activity into human motion representations. The dataset consists of 12.5 hours of synchronized video and surface electromyography (sEMG) data of 10 subjects performing various exercises. Using this dataset, we learn a bidirectional representation that predicts muscle activation from video, and conversely, reconstructs motion from muscle activation. We evaluate our model on in-distribution subjects and exercises, as well as on out-of-distribution subjects and exercises. We demonstrate how advances in modeling both modalities jointly can serve as conditioning for muscularly consistent motion generation. Putting muscles into computer vision systems will enable richer models of virtual humans, with applications in sports, fitness, and AR/VR.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 75 references that are in the map. First posted on arXiv 5 Dec 2022.
Builds on
38 of 75 references.
References · 38
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2017The Kinetics Human Action Video Datasetcited by 1,909
- 2022Human Motion Diffusion Modelcited by 1,080
- 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
- 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
- 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 620
- 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
- 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
- 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
- 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
26 more
- 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
- 2017Look, Listen and Learncited by 458
- 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 441
- 2018Video-to-Video Synthesiscited by 390
- 2017On human motion prediction using recurrent neural networkscited by 338
- 2018Everybody Dance Nowcited by 315
- 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
- 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
- 2021Robust Motion In-betweeningcited by 229
- 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
- 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 153
- 2015Visually Indicated Soundscited by 128
- 2015DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimationcited by 117
- 2018Human Motion Prediction via Spatio-Temporal Inpaintingcited by 114
- 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 114
- 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
- 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
- 2022Back to MLP: A Simple Baseline for Human Motion Predictioncited by 100
- 2020Foley Music: Learning to Generate Music from Videoscited by 83
- 2019Oops! Predicting Unintentional Action in Videocited by 82
- 2016Ambient Sound Provides Supervision for Visual Learningcited by 78
- 2020Convolutional Autoencoders for Human Motion Infillingcited by 73
- 2021Single-Shot Motion Completion with Transformercited by 59
- 2021Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structurecited by 53
- 2018Recurrent Transition Networks for Character Locomotioncited by 52
- 2018Deep Video-Based Performance Cloningcited by 32
Built on by
Not cited yet.