# Self-supervised contrastive learning performs non-linear system identification

arXiv: https://arxiv.org/abs/2410.14673 · HTML: https://mlsanity.com/paper/2410.14673/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 68 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [CITRIS: Causal Identifiability from Temporal Intervened Sequences](https://mlsanity.com/paper/2202.03169/) (2022)
- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [On Linear Identifiability of Learned Representations](https://mlsanity.com/paper/2007.00810/) (2020)
- [Learning Temporally Causal Latent Processes from General Temporal Data](https://mlsanity.com/paper/2110.05428/) (2021)
- [Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)](https://mlsanity.com/paper/2001.04872/) (2020)
- [Learning and Leveraging World Models in Visual Representation Learning](https://mlsanity.com/paper/2403.00504/) (2024)
- [Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA](https://mlsanity.com/paper/2106.09620/) (2021)
- [ODEFormer: Symbolic Regression of Dynamical Systems with Transformers](https://mlsanity.com/paper/2310.05573/) (2023)
- [Temporally Disentangled Representation Learning](https://mlsanity.com/paper/2210.13647/) (2022)

## Built on by (most cited first, 9 of 9)

- [Understanding Self-Supervised Learning via Latent Distribution Matching](https://mlsanity.com/paper/2605.03517/) (2026)
- [Maximum Likelihood Learning of Latent Dynamics Without Reconstruction](https://mlsanity.com/paper/2505.23569/) (2025)
- [Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions](https://mlsanity.com/paper/2510.21706/) (2025)
- [Extracting Governing Equations from Latent Dynamics via Multi-View Contrastive Learning](https://mlsanity.com/paper/2606.13260/) (2026)
- [ReNF: Rethinking the Design of Neural Long-Term Time Series Forecasters](https://mlsanity.com/paper/2509.25914/) (2025)
- [Identify then Realize: Contrastive Learning of Latent Port-Hamiltonian Dynamics from Partial Observations](https://mlsanity.com/paper/2605.16682/) (2026)
- [Template Collapse and Information-Theoretic Limits in Camera rPPG Pulse Morphology Restoration](https://mlsanity.com/paper/2606.03802/) (2026)
- [Machine Learning Methods for Studying Latent Neural Activity Dynamics](https://mlsanity.com/paper/2606.10530/) (2026)
- [Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisance](https://mlsanity.com/paper/2609.37789/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) approaches have brought tremendous success across many tasks and domains. It has been argued that these successes can be attributed to a link between SSL and identifiable representation learning: Temporal structure and auxiliary variables ensure that latent representations are related to the true underlying generative factors of the data. Here, we deepen this connection and show that SSL can perform system identification in latent space. We propose dynamics contrastive learning, a framework to uncover linear, switching linear and non-linear dynamics under a non-linear observation model, give theoretical guarantees and validate them empirically.