mlsanityData to 7 Oct 2026

Self-supervised contrastive learning performs non-linear system identification

arXiv 2410.14673 ↗First posted 18 Oct 2024stat.ML cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) approaches have brought tremendous success across many tasks and domains. It has been argued that these successes can be attributed to a link between SSL and identifiable representation learning: Temporal structure and auxiliary variables ensure that latent representations are related to the true underlying generative factors of the data. Here, we deepen this connection and show that SSL can perform system identification in latent space. We propose dynamics contrastive learning, a framework to uncover linear, switching linear and non-linear dynamics under a non-linear observation model, give theoretical guarantees and validate them empirically.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 43 of its 68 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

43 of 68 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2018World Modelscited by 1,444
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
31 more
  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  4. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  5. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  6. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  7. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
  8. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  9. 2022End-to-end symbolic regression with transformerscited by 195
  10. 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 132
  11. 2022Learnable latent embeddings for joint behavioral and neural analysiscited by 113
  12. 2020On Linear Identifiability of Learned Representationscited by 103
  13. 2021Learning Temporally Causal Latent Processes from General Temporal Datacited by 96
  14. 2020Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)cited by 91
  15. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 71
  16. 2021Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICAcited by 64
  17. 2022Temporally Disentangled Representation Learningcited by 59
  18. 2023ODEFormer: Symbolic Regression of Dynamical Systems with Transformerscited by 58
  19. 2023Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learningcited by 48
  20. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  21. 2016Linear dynamical neural population models through nonlinear embeddingscited by 36
  22. 2019Learning interpretable continuous-time models of latent stochastic dynamical systemscited by 29
  23. 2019Collapsed Amortized Variational Inference for Switching Nonlinear Dynamical Systemscited by 26
  24. 2023Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysiscited by 21
  25. 2021Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systemscited by 17
  26. 2023Towards a Unified Framework of Contrastive Learning for Disentangled Representationscited by 13
  27. 2021Discovering State Variables Hidden in Experimental Datacited by 12
  28. 2021Building population models for large-scale neural recordings: opportunities and pitfallscited by 9
  29. 2023On the Identifiability of Switching Dynamical Systemscited by 9
  30. 2019Explicit-Duration Markov Switching Modelscited by 3
  31. 2021Segmenting Hybrid Trajectories using Latent ODEscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2026Understanding Self-Supervised Learning via Latent Distribution Matchingcited by 4
  2. 2025Maximum Likelihood Learning of Latent Dynamics Without Reconstructioncited by 3
  3. 2025Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actionscited by 2
  4. 2026Extracting Governing Equations from Latent Dynamics via Multi-View Contrastive Learningcited by 1
  5. 2025ReNF: Rethinking the Design of Neural Long-Term Time Series Forecasterscited by 0
  6. 2026Identify then Realize: Contrastive Learning of Latent Port-Hamiltonian Dynamics from Partial Observationscited by 0
  7. 2026Template Collapse and Information-Theoretic Limits in Camera rPPG Pulse Morphology Restorationcited by 0
  8. 2026Machine Learning Methods for Studying Latent Neural Activity Dynamicscited by 0
  9. 2026Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisancecited by 0