# Contrastive Learning for Multimodal Human Activity Recognition with Limited Labeled Data

arXiv: https://arxiv.org/abs/2604.23281 · HTML: https://mlsanity.com/paper/2604.23281/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 25 Apr 2026.

## Builds on (most cited first, 7 of 7)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [A Survey on Generative Adversarial Networks: Variants, Applications, and Training](https://mlsanity.com/paper/2006.05132/) (2020)
- [Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2012.05333/) (2020)
- [CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking](https://mlsanity.com/paper/2307.16847/) (2023)
- [MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels](https://mlsanity.com/paper/2404.01958/) (2024)
- [Babel: A Scalable Pre-trained Model for Multi-Modal Sensing via Expandable Modality Alignment](https://mlsanity.com/paper/2407.17777/) (2024)

## Abstract (arXiv)

Human activity recognition serves as the foundation for various emerging applications. In recent years, researchers have used collaborative sensing of multi-source sensors to capture complex and dynamic human activities. However, multimodal human activity sensing typically encounters highly heterogeneous data across modalities and label scarcity, resulting in an application gap between existing solutions and real-world needs. In this paper, we propose CLMM, a general contrastive learning framework for human activity recognition that achieves effective multimodal recognition with limited labeled data. CLMM employs a novel two-stage training strategy. In the first stage, CLMM employs a CNN-DiffTransformer encoder to capture cross-modal shared information by extracting local and global features. Meanwhile, a hard-positive samples weighting algorithm enhances gradient propagation to reinforce shared learning. In the second stage, a dual-branch architecture combining quality-guided attention and bidirectional gated units captures modality-specific information, while a primary-auxiliary collaborative training strategy fuses both shared and modality-specific information. Experimental results on three public datasets demonstrate that CLMM significantly improves state-of-the-art baselines in both recognition accuracy and convergence performance.