# A theoretical framework for self-supervised contrastive learning for continuous dependent data

arXiv: https://arxiv.org/abs/2506.09785 · HTML: https://mlsanity.com/paper/2506.09785/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 56 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting](https://mlsanity.com/paper/2202.01575/) (2022)
- [OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning](https://mlsanity.com/paper/2306.11249/) (2023)
- [The Power of Contrast for Feature Learning: A Theoretical Analysis](https://mlsanity.com/paper/2110.02473/) (2021)
- [TimesURL: Self-supervised Contrastive Learning for Universal Time Series Representation Learning](https://mlsanity.com/paper/2312.15709/) (2023)
- [Soft Contrastive Learning for Time Series](https://mlsanity.com/paper/2312.16424/) (2023)
- [Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Least](https://mlsanity.com/paper/2302.09195/) (2023)
- [Correlated Time Series Self-Supervised Representation Learning via Spatiotemporal Bootstrapping](https://mlsanity.com/paper/2306.06994/) (2023)
- [AtmoDist: Self-supervised Representation Learning for Atmospheric Dynamics](https://mlsanity.com/paper/2202.01897/) (2022)
- [Similarity learning for wells based on logging data](https://mlsanity.com/paper/2202.05583/) (2022)
- [Non-contrastive representation learning for intervals from well logs](https://mlsanity.com/paper/2209.14750/) (2022)
- [Improving Time Series Encoding with Noise-Aware Self-Supervised Learning and an Efficient Encoder](https://mlsanity.com/paper/2306.06579/) (2023)
- [EMIT- Event-Based Masked Auto Encoding for Irregular Time Series](https://mlsanity.com/paper/2409.16554/) (2024)

## Abstract (arXiv)

Self-supervised learning (SSL) has emerged as a powerful approach to learning representations, particularly in the field of computer vision. However, its application to dependent data, such as temporal and spatio-temporal domains, remains underexplored. Besides, traditional contrastive SSL methods often assume semantic independence between samples, which does not hold for dependent data exhibiting complex correlations. We propose a novel theoretical framework for contrastive SSL tailored to continuous dependent data, which allows the nearest samples to be semantically close to each other. In particular, we propose two possible textitground truth similarity measures between objects -- hard and soft closeness. Under it, we derive an analytical form for the textitestimated similarity matrix that accommodates both types of closeness between samples, thereby introducing dependency-aware loss functions. We validate our approach, Dependent TS2Vec, on temporal and spatio-temporal downstream problems. Given the dependency patterns presented in the data, our approach surpasses modern ones for dependent data, highlighting the effectiveness of our theoretically grounded loss functions for SSL in capturing spatio-temporal dependencies. Specifically, we outperform TS2Vec on the standard UEA and UCR benchmarks, with accuracy improvements of 4.17% and 2.08%, respectively. Furthermore, on the drought classification task, which involves complex spatio-temporal patterns, our method achieves a 7% higher ROC-AUC score.