# Contrastive learning of strong-mixing continuous-time stochastic processes

arXiv: https://arxiv.org/abs/2103.02740 · HTML: https://mlsanity.com/paper/2103.02740/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Contrastive learning, multi-view redundancy, and linear models](https://mlsanity.com/paper/2008.10150/) (2020)
- [Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases](https://mlsanity.com/paper/2007.13916/) (2020)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Flow Contrastive Estimation of Energy-Based Models](https://mlsanity.com/paper/1912.00589/) (2019)
- [Contrastive estimation reveals topic posterior information to linear models](https://mlsanity.com/paper/2003.02234/) (2020)
- [Sampling from a log-concave distribution with Projected Langevin Monte Carlo](https://mlsanity.com/paper/1507.02564/) (2015)

## Built on by (most cited first, 3 of 3)

- [Language modeling via stochastic processes](https://mlsanity.com/paper/2203.11370/) (2022)
- [Leveraging Time Irreversibility with Order-Contrastive Pre-training](https://mlsanity.com/paper/2111.02599/) (2021)
- [BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation](https://mlsanity.com/paper/2405.17764/) (2024)

## Abstract (arXiv)

Contrastive learning is a family of self-supervised methods where a model is trained to solve a classification task constructed from unlabeled data. It has recently emerged as one of the leading learning paradigms in the absence of labels across many different domains (e.g. brain imaging, text, images). However, theoretical understanding of many aspects of training, both statistical and algorithmic, remain fairly elusive. In this work, we study the setting of time series -- more precisely, when we get data from a strong-mixing continuous-time stochastic process. We show that a properly constructed contrastive learning task can be used to estimate the transition kernel for small-to-mid-range intervals in the diffusion case. Moreover, we give sample complexity bounds for solving this task and quantitatively characterize what the value of the contrastive loss implies for distributional closeness of the learned kernel. As a byproduct, we illuminate the appropriate settings for the contrastive distribution, as well as other hyperparameters in this setup.