# CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking

arXiv: https://arxiv.org/abs/2307.16847 · HTML: https://mlsanity.com/paper/2307.16847/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 20 of 20)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review](https://mlsanity.com/paper/2304.09355/) (2023)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Time Series Change Point Detection with Self-Supervised Contrastive Predictive Coding](https://mlsanity.com/paper/2011.14097/) (2020)
- [Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2012.05333/) (2020)
- [Self-supervised transfer learning of physiological representations from free-living wearable data](https://mlsanity.com/paper/2011.12121/) (2020)
- [Sense and Learn: Self-Supervision for Omnipresent Sensors](https://mlsanity.com/paper/2009.13233/) (2020)
- [Exploring Self-Supervised Representation Ensembles for COVID-19 Cough Classification](https://mlsanity.com/paper/2105.07566/) (2021)

## Built on by (most cited first, 19 of 19)

- [Scaling Wearable Foundation Models](https://mlsanity.com/paper/2410.13638/) (2024)
- [SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition](https://mlsanity.com/paper/2410.10624/) (2024)
- [Wearable Accelerometer Foundation Models for Health via Knowledge Distillation](https://mlsanity.com/paper/2412.11276/) (2024)
- [PRIMUS: Pretraining IMU Encoders with Multimodal Self-Supervision](https://mlsanity.com/paper/2411.15127/) (2024)
- [ViLCo-Bench: VIdeo Language COntinual learning Benchmark](https://mlsanity.com/paper/2406.13123/) (2024)
- [COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition](https://mlsanity.com/paper/2503.07259/) (2025)
- [OTIS: Learning High-Quality Time Series Features With Tiny Encoders](https://mlsanity.com/paper/2410.07299/) (2024)
- [Time2Lang: Bridging Time-Series Foundation Models and Large Language Models for Health Sensing Beyond Prompting](https://mlsanity.com/paper/2502.07608/) (2025)
- [Towards Generalizable Human Activity Recognition: A Survey](https://mlsanity.com/paper/2508.12213/) (2025)
- [From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signals](https://mlsanity.com/paper/2405.16083/) (2024)
- [Toward Foundational Model for Sleep Analysis Using a Multimodal Hybrid Self-Supervised Learning Framework](https://mlsanity.com/paper/2502.17481/) (2025)
- [Representation Learning for Wearable-Based Applications in the Case of Missing Data](https://mlsanity.com/paper/2401.05437/) (2024)
- [SlotFM: A Motion Foundation Model with Slot Attention for Diverse Downstream Tasks](https://mlsanity.com/paper/2509.21673/) (2025)
- [PhysioME: A Robust Multimodal Self-Supervised Framework for Physiological Signals with Missing Modalities](https://mlsanity.com/paper/2510.11110/) (2025)
- [Prototype-based Self-Supervised Multimodal Learning for PPG and Accelerometry Signals](https://mlsanity.com/paper/2510.09764/) (2025)
- [Physiology as Language: Translating Respiration to Sleep EEG](https://mlsanity.com/paper/2602.00526/) (2026)
- [Multi-Station WiFi CSI Sensing Framework Robust to Station-wise Feature Missingness and Limited Labeled Data](https://mlsanity.com/paper/2603.11858/) (2026)
- [Contrastive Learning for Multimodal Human Activity Recognition with Limited Labeled Data](https://mlsanity.com/paper/2604.23281/) (2026)
- [LiteEMG-FM: An Efficient and Deployable Foundation Model for Robust EMG Sensing](https://mlsanity.com/paper/2610.02497/) (2026)

## Abstract (arXiv)

Limited availability of labeled data for machine learning on multimodal time-series extensively hampers progress in the field. Self-supervised learning (SSL) is a promising approach to learning data representations without relying on labels. However, existing SSL methods require expensive computations of negative pairs and are typically designed for single modalities, which limits their versatility. We introduce CroSSL (Cross-modal SSL), which puts forward two novel concepts: masking intermediate embeddings produced by modality-specific encoders, and their aggregation into a global embedding through a cross-modal aggregator that can be fed to down-stream classifiers. CroSSL allows for handling missing modalities and end-to-end cross-modal learning without requiring prior data preprocessing for handling missing inputs or negative-pair sampling for contrastive learning. We evaluate our method on a wide range of data, including motion sensors such as accelerometers or gyroscopes and biosignals (heart rate, electroencephalograms, electromyograms, electrooculograms, and electrodermal) to investigate the impact of masking ratios and masking strategies for various data types and the robustness of the learned representations to missing data. Overall, CroSSL outperforms previous SSL and supervised benchmarks using minimal labeled data, and also sheds light on how latent masking can improve cross-modal learning. Our code is open-sourced at https://github.com/dr-bell/CroSSL.