mlsanityData to 14 Oct 2026

IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text

arXiv 2210.14395 ↗First posted 26 Oct 2022cs.CV cs.CL cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos -- while preserving the transitivity across these modalities. We explore several new IMU-based applications that IMU2CLIP enables, such as motion-based media retrieval and natural language reasoning tasks with motion data. In addition, we show that IMU2CLIP can significantly improve the downstream performance when fine-tuned for each application (e.g. activity recognition), demonstrating the universal usage of IMU2CLIP as a new pre-trained resource. Our code will be made publicly available.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 11 of its 23 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

11 of 23 references.

References · 11

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  10. 2022Egocentric Video-Language Pretrainingcited by 232
  11. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 16; 2025: 8; 2026: 6.

37 papers.

Sort:

By area:

  1. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  2. 2024Large Language Models for Time Series: A Surveycited by 114
  3. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  4. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  5. 2023The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Modelscited by 34
  6. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  7. 2024UniMTS: Unified Pre-training for Motion Time Seriescited by 26
  8. 2024Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalitiescited by 24
  9. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  10. 2024Wearable Accelerometer Foundation Models for Health via Knowledge Distillationcited by 16
  11. 2024Past, Present, and Future of Sensor-Based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Taskcited by 15
  12. 2026TSAQA: Time Series Analysis Question And Answering Benchmarkcited by 11
25 more
  1. 2024AccidentGPT: Large Multi-Modal Foundation Model for Traffic Accident Analysiscited by 10
  2. 2024A Survey on Multimodal Wearable Sensor-based Human Action Recognitioncited by 10
  3. 2024Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Themcited by 10
  4. 2024PRIMUS: Pretraining IMU Encoders with Multimodal Self-Supervisioncited by 10
  5. 2024CLaSP: Learning Concepts for Time-Series Signals from Natural Language Supervisioncited by 6
  6. 2024Spherical World-Locking for Audio-Visual Localization in Egocentric Videoscited by 5
  7. 2023Multimodal Contrastive Learning with Hard Negative Sampling for Human Activity Recognitioncited by 4
  8. 2024A Survey of IMU Based Cross-Modal Transfer Learning in Human Activity Recognitioncited by 4
  9. 2025Mojito: LLM-Aided Motion Instructor with Jitter-Reduced Inertial Tokenscited by 4
  10. 2023Selecting the motion ground truth for loose-fitting wearables: benchmarking optical MoCap methodscited by 3
  11. 2025EgoCHARM: Resource-Efficient Hierarchical Activity Recognition using an Egocentric IMU Sensorcited by 3
  12. 2023Contrastive Left-Right Wearable Sensors (IMUs) Consistency Matching for HARcited by 2
  13. 2023Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semanticscited by 2
  14. 2024MuJo: Multimodal Joint Feature Space Learning for Human Activity Recognitioncited by 2
  15. 2024C3T: Cross-modal Transfer Through Time for Sensor-based Human Activity Recognitioncited by 2
  16. 2026EgoSelf: From Memory to Personalized Egocentric Assistantcited by 2
  17. 2025From Motion Signals to Insights: A Unified Framework for Student Behavior Analysis and Feedback in Physical Education Classescited by 1
  18. 2026MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignmentcited by 1
  19. 2025Towards Achieving Perfect Multimodal Alignmentcited by 0
  20. 2025SImpHAR: Advancing impedance-based human activity recognition using 3D simulation and text-to-motion modelscited by 0
  21. 2025Prototype-based Self-Supervised Multimodal Learning for PPG and Accelerometry Signalscited by 0
  22. 2025DETACH : Decomposed Spatio-Temporal Alignment for Exocentric Video and Ambient Sensors with Staged Learningcited by 0
  23. 2026BasketHAR: A Multimodal Dataset for Human Activity Recognition and Sport Analysis in Basketball Training Scenarioscited by 0
  24. 2026Modular Retrieval-Augmented Generalization for Human Action Recognitioncited by 0
  25. 2026CHEAR: Confusion-Hierarchy Evidence-Aware Reasoning for Cost-Efficient Human Activity Recognitioncited by 0