# On Contrastive Representations of Stochastic Processes

arXiv: https://arxiv.org/abs/2106.10052 · HTML: https://mlsanity.com/paper/2106.10052/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 18 Jun 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [On Contrastive Learning for Likelihood-free Inference](https://mlsanity.com/paper/2002.03712/) (2020)
- [Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions](https://mlsanity.com/paper/1710.10304/) (2017)
- [Function Contrastive Learning of Transferable Meta-Representations](https://mlsanity.com/paper/2010.07093/) (2020)
- [A Simple Framework for Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2010.02038/) (2020)
- [Improving Transformation Invariance in Contrastive Representation Learning](https://mlsanity.com/paper/2010.09515/) (2020)
- [Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings](https://mlsanity.com/paper/1906.02236/) (2019)

## Built on by (most cited first, 12 of 12)

- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [Latent Bottlenecked Attentive Neural Processes](https://mlsanity.com/paper/2211.08458/) (2022)
- [The Neural Process Family: Survey, Applications and Perspectives](https://mlsanity.com/paper/2209.00517/) (2022)
- [Contrastive Conditional Neural Processes](https://mlsanity.com/paper/2203.03978/) (2022)
- [Deep Stochastic Processes via Functional Markov Transition Operators](https://mlsanity.com/paper/2305.15574/) (2023)
- [Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder](https://mlsanity.com/paper/2310.16318/) (2023)
- [Spectral Convolutional Conditional Neural Processes](https://mlsanity.com/paper/2404.13182/) (2024)
- [Adaptive Conditional Quantile Neural Processes](https://mlsanity.com/paper/2305.18777/) (2023)
- [Learning to Learn with Contrastive Meta-Objective](https://mlsanity.com/paper/2410.05975/) (2024)
- [Adversarially Contrastive Estimation of Conditional Neural Processes](https://mlsanity.com/paper/2303.13004/) (2023)
- [Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes](https://mlsanity.com/paper/2308.00858/) (2023)
- [BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation](https://mlsanity.com/paper/2405.17764/) (2024)

## Abstract (arXiv)

Learning representations of stochastic processes is an emerging problem in machine learning with applications from meta-learning to physical object models to time series. Typical methods rely on exact reconstruction of observations, but this approach breaks down as observations become high-dimensional or noise distributions become complex. To address this, we propose a unifying framework for learning contrastive representations of stochastic processes (CReSP) that does away with exact reconstruction. We dissect potential use cases for stochastic process representations, and propose methods that accommodate each. Empirically, we show that our methods are effective for learning representations of periodic functions, 3D objects and dynamical processes. Our methods tolerate noisy high-dimensional observations better than traditional approaches, and the learned representations transfer to a range of downstream tasks.