# Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Series

arXiv: https://arxiv.org/abs/2203.09270 · HTML: https://mlsanity.com/paper/2203.09270/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [Self-supervised representation learning from electroencephalography signals](https://mlsanity.com/paper/1911.05419/) (2019)
- [Adaptive Exploration for Unsupervised Person Re-Identification](https://mlsanity.com/paper/1907.04194/) (2019)
- [Learning to Exploit Invariances in Clinical Time-Series Data using Sequence Transformer Networks](https://mlsanity.com/paper/1808.06725/) (2018)

## Built on by (most cited first, 30 of 54)

- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series](https://mlsanity.com/paper/2310.14017/) (2023)
- [Soft Contrastive Learning for Time Series](https://mlsanity.com/paper/2312.16424/) (2023)
- [Learning to Embed Time Series Patches Independently](https://mlsanity.com/paper/2312.16427/) (2023)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Toward a Foundation Model for Time Series Data](https://mlsanity.com/paper/2310.03916/) (2023)
- [Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals](https://mlsanity.com/paper/2309.05927/) (2023)
- [TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling](https://mlsanity.com/paper/2402.02475/) (2024)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [Label-efficient Time Series Representation Learning: A Review](https://mlsanity.com/paper/2302.06433/) (2023)
- [Unsupervised Representation Learning for Time Series: A Review](https://mlsanity.com/paper/2308.01578/) (2023)
- [Hierarchical Pretraining on Multimodal Electronic Health Records](https://mlsanity.com/paper/2310.07871/) (2023)
- [Look Into the LITE in Deep Learning for Time Series Classification](https://mlsanity.com/paper/2409.02869/) (2024)
- [A Co-training Approach for Noisy Time Series Learning](https://mlsanity.com/paper/2308.12551/) (2023)
- [Series2Vec: Similarity-based Self-supervised Representation Learning for Time Series Classification](https://mlsanity.com/paper/2312.03998/) (2023)
- [Joint Contrastive Learning with Feature Alignment for Cross-Corpus EEG-based Emotion Recognition](https://mlsanity.com/paper/2404.09559/) (2024)
- [Guidelines for Augmentation Selection in Contrastive Learning for Time Series Classification](https://mlsanity.com/paper/2407.09336/) (2024)
- [TopoCL: Topological Contrastive Learning for Time Series](https://mlsanity.com/paper/2502.02924/) (2025)
- [A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospects](https://mlsanity.com/paper/2506.19769/) (2025)
- [A clinically motivated self-supervised approach for content-based image retrieval of CT liver images](https://mlsanity.com/paper/2207.04812/) (2022)
- [From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signals](https://mlsanity.com/paper/2405.16083/) (2024)
- [Contrastive Learning Is Not Optimal for Quasiperiodic Time Series](https://mlsanity.com/paper/2407.17073/) (2024)
- [Multi-View Contrastive Learning for Robust Domain Adaptation in Medical Time Series Analysis](https://mlsanity.com/paper/2506.22393/) (2025)
- [Self-supervised learning-based general laboratory progress pretrained model for cardiovascular event detection](https://mlsanity.com/paper/2303.06980/) (2023)
- [mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design](https://mlsanity.com/paper/2408.09048/) (2024)

## Abstract (arXiv)

The lack of labeled data is a key challenge for learning useful representation from time series data. However, an unsupervised representation framework that is capable of producing high quality representations could be of great value. It is key to enabling transfer learning, which is especially beneficial for medical applications, where there is an abundance of data but labeling is costly and time consuming. We propose an unsupervised contrastive learning framework that is motivated from the perspective of label smoothing. The proposed approach uses a novel contrastive loss that naturally exploits a data augmentation scheme in which new samples are generated by mixing two data samples with a mixing component. The task in the proposed framework is to predict the mixing component, which is utilized as soft targets in the loss function. Experiments demonstrate the framework's superior performance compared to other representation learning approaches on both univariate and multivariate time series and illustrate its benefits for transfer learning for clinical time series.