# COCOA: Cross Modality Contrastive Learning for Sensor Data

arXiv: https://arxiv.org/abs/2208.00467 · HTML: https://mlsanity.com/paper/2208.00467/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 94 references that are in the map.
- First posted on arXiv 31 Jul 2022.

## Builds on (most cited first, 30 of 68)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)

## Built on by (most cited first, 21 of 21)

- [Evaluating Large Language Models as Virtual Annotators for Time-series Physical Sensing Data](https://mlsanity.com/paper/2403.01133/) (2024)
- [Past, Present, and Future of Sensor-Based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task](https://mlsanity.com/paper/2411.14452/) (2024)
- [LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms](https://mlsanity.com/paper/2311.11420/) (2023)
- [MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels](https://mlsanity.com/paper/2404.01958/) (2024)
- [Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors](https://mlsanity.com/paper/2410.20034/) (2024)
- [COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition](https://mlsanity.com/paper/2503.07259/) (2025)
- [Multi-view self-supervised learning for multivariate variable-channel time series](https://mlsanity.com/paper/2307.09614/) (2023)
- [Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2211.06173/) (2022)
- [Weak-Annotation of HAR Datasets using Vision Foundation Models](https://mlsanity.com/paper/2408.05169/) (2024)
- [Suite-IN: Aggregating Motion Features from Apple Suite for Robust Inertial Navigation](https://mlsanity.com/paper/2411.07828/) (2024)
- [InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals](https://mlsanity.com/paper/2504.09707/) (2025)
- [Subject Invariant Contrastive Learning for Human Activity Recognition](https://mlsanity.com/paper/2507.03250/) (2025)
- [SudokuSens: Enhancing Deep Learning Robustness for IoT Sensing Applications using a Generative Approach](https://mlsanity.com/paper/2402.02275/) (2024)
- [SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learning](https://mlsanity.com/paper/2405.00712/) (2024)
- [Self-supervised Activity Representation Learning with Incremental Data: An Empirical Study](https://mlsanity.com/paper/2305.00619/) (2023)
- [SelfReplay: Adapting Self-Supervised Sensory Models via Adaptive Meta-Task Replay](https://mlsanity.com/paper/2404.15305/) (2024)
- [Auto-Augmentation Contrastive Learning for Wearable-based Human Activity Recognition](https://mlsanity.com/paper/2602.02542/) (2026)
- [MassNet: A Deep Learning Approach for Body Weight Extraction from A Single Pressure Image](https://mlsanity.com/paper/2303.10136/) (2023)
- [Contrastive Learning-based User Identification with Limited Data on Smart Textiles](https://mlsanity.com/paper/2409.07488/) (2024)
- [SitLLM: Large Language Models for Sitting Posture Health Understanding via Pressure Sensor Data](https://mlsanity.com/paper/2509.12994/) (2025)
- [Frequency-Domain Multi-Modality Transportation Modeling](https://mlsanity.com/paper/2607.08475/) (2026)

## Abstract (arXiv)

Self-Supervised Learning (SSL) is a new paradigm for learning discriminative representations without labelled data and has reached comparable or even state-of-the-art results in comparison to supervised counterparts. Contrastive Learning (CL) is one of the most well-known approaches in SSL that attempts to learn general, informative representations of data. CL methods have been mostly developed for applications in computer vision and natural language processing where only a single sensor modality is used. A majority of pervasive computing applications, however, exploit data from a range of different sensor modalities. While existing CL methods are limited to learning from one or two data sources, we propose COCOA (Cross mOdality COntrastive leArning), a self-supervised model that employs a novel objective function to learn quality representations from multisensor data by computing the cross-correlation between different data modalities and minimizing the similarity between irrelevant instances. We evaluate the effectiveness of COCOA against eight recently introduced state-of-the-art self-supervised models, and two supervised baselines across five public datasets. We show that COCOA achieves superior classification performance to all other approaches. Also, COCOA is far more label-efficient than the other baselines including the fully supervised model using only one-tenth of available labelled data.