# Time-Series Representation Learning via Temporal and Contextual Contrasting

arXiv: https://arxiv.org/abs/2106.14112 · HTML: https://mlsanity.com/paper/2106.14112/

## Key facts

- Cited by 345 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 26 Jun 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [Subject-Aware Contrastive Learning for Biosignals](https://mlsanity.com/paper/2007.04871/) (2020)
- [Adversarial Unsupervised Representation Learning for Activity Time-Series](https://mlsanity.com/paper/1811.06847/) (2018)

## Built on by (most cited first, 30 of 345)

- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2412.07236/) (2024)
- [Deep Time Series Models: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2407.13278/) (2024)
- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting](https://mlsanity.com/paper/2202.01575/) (2022)
- [TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2306.09364/) (2023)
- [LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters](https://mlsanity.com/paper/2308.08469/) (2023)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion](https://mlsanity.com/paper/2202.04770/) (2022)
- [ECG-FM: An Open Electrocardiogram Foundation Model](https://mlsanity.com/paper/2408.05178/) (2024)
- [TSLANet: Rethinking Transformers for Time Series Representation Learning](https://mlsanity.com/paper/2404.08472/) (2024)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement](https://mlsanity.com/paper/2403.06659/) (2024)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting](https://mlsanity.com/paper/2403.09898/) (2024)
- [Ti-MAE: Self-Supervised Masked Time Series Autoencoders](https://mlsanity.com/paper/2301.08871/) (2023)
- [Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series](https://mlsanity.com/paper/2310.14017/) (2023)
- [Time Series Contrastive Learning with Information-Aware Augmentations](https://mlsanity.com/paper/2303.11911/) (2023)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [TimesURL: Self-supervised Contrastive Learning for Universal Time Series Representation Learning](https://mlsanity.com/paper/2312.15709/) (2023)
- [EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model](https://mlsanity.com/paper/2401.10278/) (2024)
- [Soft Contrastive Learning for Time Series](https://mlsanity.com/paper/2312.16424/) (2023)
- [EEG2Rep: Enhancing Self-supervised EEG Representation Through Informative Masked Inputs](https://mlsanity.com/paper/2402.17772/) (2024)
- [Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer](https://mlsanity.com/paper/2205.09928/) (2022)
- [ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data](https://mlsanity.com/paper/2203.08321/) (2022)
- [TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders](https://mlsanity.com/paper/2303.00320/) (2023)

## Abstract (arXiv)

Learning decent representations from unlabeled time-series data with temporal dynamics is a very challenging task. In this paper, we propose an unsupervised Time-Series representation learning framework via Temporal and Contextual Contrasting (TS-TCC), to learn time-series representation from unlabeled data. First, the raw time-series data are transformed into two different yet correlated views by using weak and strong augmentations. Second, we propose a novel temporal contrasting module to learn robust temporal representations by designing a tough cross-view prediction task. Last, to further learn discriminative representations, we propose a contextual contrasting module built upon the contexts from the temporal contrasting module. It attempts to maximize the similarity among different contexts of the same sample while minimizing similarity among contexts of different samples. Experiments have been carried out on three real-world time-series datasets. The results manifest that training a linear classifier on top of the features learned by our proposed TS-TCC performs comparably with the supervised training. Additionally, our proposed TS-TCC shows high efficiency in few-labeled data and transfer learning scenarios. The code is publicly available at https://github.com/emadeldeen24/TS-TCC.