mlsanityData to 7 Oct 2026

Learning Joint Representation of Human Motion and Language

arXiv 2210.15187 ↗First posted 27 Oct 2022cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

In this work, we present MoLang (a Motion-Language connecting model) for learning joint representation of human motion and language, leveraging both unpaired and paired datasets of motion and language modalities. To this end, we propose a motion-language model with contrastive learning, empowering our model to learn better generalizable representations of the human motion domain. Empirical results show that our model learns strong representations of human motion data through navigating language modality. Our proposed method is able to perform both action recognition and motion retrieval tasks with a single model where it outperforms state-of-the-art approaches on a number of action recognition benchmarks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

43 of 55 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  10. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  11. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  12. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
31 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  2. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  3. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  4. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  5. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  6. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  7. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  8. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  9. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  10. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  11. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  12. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 339
  13. 2017On human motion prediction using recurrent neural networkscited by 336
  14. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 317
  15. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  16. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  17. 2021Robust Motion In-betweeningcited by 223
  18. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  19. 2016Movie Descriptioncited by 180
  20. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  21. 2016Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognitioncited by 142
  22. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  23. 2016Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networkscited by 63
  24. 2021Single-Shot Motion Completion with Transformercited by 59
  25. 2019Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modelingcited by 32
  26. 2021Multimodal Self-Supervised Learning of General Audio Representationscited by 31
  27. 2017What Actions are Needed for Understanding Human Actions in Videos?cited by 29
  28. 20203D Human Shape Reconstruction from a Polarization Imagecited by 28
  29. 2020DOPE: Distillation Of Part Experts for whole-body 3D pose estimation in the wildcited by 25
  30. 2022Conditional Motion In-betweeningcited by 25
  31. 2019A Large-scale Varying-view RGB-D Action Dataset for Arbitrary-view Human Action Recognitioncited by 16

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2023Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Languagecited by 11
  2. 2024MLP: Motion Label Prior for Temporal Sentence Localization in Untrimmed 3D Human Motionscited by 3
  3. 2024Joint-Dataset Learning and Cross-Consistent Regularization for Text-to-Motion Retrievalcited by 1