# PRIMUS: Pretraining IMU Encoders with Multimodal Self-Supervision

arXiv: https://arxiv.org/abs/2411.15127 · HTML: https://mlsanity.com/paper/2411.15127/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [Exploring Contrastive Learning in Human Activity Recognition for Healthcare](https://mlsanity.com/paper/2011.11542/) (2020)
- [Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data](https://mlsanity.com/paper/2206.02909/) (2022)
- [IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text](https://mlsanity.com/paper/2210.14395/) (2022)
- [CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking](https://mlsanity.com/paper/2307.16847/) (2023)
- [CoughTrigger: Earbuds IMU Based Cough Detection Activator Using An Energy-efficient Sensitivity-prioritized Time Series Classifier](https://mlsanity.com/paper/2111.04185/) (2021)

## Built on by (most cited first, 10 of 10)

- [Towards Generalizable Human Activity Recognition: A Survey](https://mlsanity.com/paper/2508.12213/) (2025)
- [Seeing without Pixels: Perception from Camera Trajectories](https://mlsanity.com/paper/2511.21681/) (2025)
- [EgoCHARM: Resource-Efficient Hierarchical Activity Recognition using an Egocentric IMU Sensor](https://mlsanity.com/paper/2504.17735/) (2025)
- [SlotFM: A Motion Foundation Model with Slot Attention for Diverse Downstream Tasks](https://mlsanity.com/paper/2509.21673/) (2025)
- [Sonata: A Hybrid World Model for Inertial Kinematics under Clinical Data Scarcity](https://mlsanity.com/paper/2604.18058/) (2026)
- [MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment](https://mlsanity.com/paper/2602.19004/) (2026)
- [DETACH : Decomposed Spatio-Temporal Alignment for Exocentric Video and Ambient Sensors with Staged Learning](https://mlsanity.com/paper/2512.20409/) (2025)
- [Modular Retrieval-Augmented Generalization for Human Action Recognition](https://mlsanity.com/paper/2605.08117/) (2026)
- [Beyond Motion Primitives: Behavioral Activity Recognition from Head-Mounted IMU](https://mlsanity.com/paper/2605.27464/) (2026)
- [IMU-Bench: Benchmarking Wrist IMU Representations Across Kinematic, Semantic and Clinical Tasks](https://mlsanity.com/paper/2610.19481/) (2026)

## Abstract (arXiv)

Sensing human motions through Inertial Measurement Units (IMUs) embedded in personal devices has enabled significant applications in health and wellness. Labeled IMU data is scarce, however, unlabeled or weakly labeled IMU data can be used to model human motions. For video or text modalities, the "pretrain and adapt" approach utilizes large volumes of unlabeled or weakly labeled data to build a strong feature extractor, followed by adaptation to specific tasks using limited labeled data. However, pretraining methods are poorly understood for IMU data, and pipelines are rarely evaluated on out-of-domain tasks. We propose PRIMUS: a method for PRetraining IMU encoderS that uses a novel pretraining objective that is empirically validated based on downstream performance on both in-domain and out-of-domain datasets. The PRIMUS objective effectively enhances downstream performance by combining self-supervision, multimodal, and nearest-neighbor supervision. With fewer than 500 labeled samples per class, PRIMUS improves test accuracy by up to 15%, compared to state-of-the-art baselines. To benefit the broader community, we have open-sourced our code at github.com/nokia-bell-labs/pretrained-imu-encoders.