mlsanityData to 7 Oct 2026

Invariance-adapted decomposition and Lasso-type contrastive learning

arXiv 2210.07413 ↗First posted 13 Oct 2022stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed the effectiveness of contrastive learning in obtaining the representation of dataset that is useful in interpretation and downstream tasks. However, the mechanism that describes this effectiveness have not been thoroughly analyzed, and many studies have been conducted to investigate the data structures captured by contrastive learning. In particular, the recent study of citetcontentisolate has shown that contrastive learning is capable of decomposing the data space into the space that is invariant to all augmentations and its complement. In this paper, we introduce the notion of invariance-adapted latent space that decomposes the data space into the intersections of the invariant spaces of each augmentation and their complements. This decomposition generalizes the one introduced in citetcontentisolate, and describes a structure that is analogous to the frequencies in the harmonic analysis of a group. We experimentally show that contrastive learning with lasso-type metric can be used to find an invariance-adapted latent space, thereby suggesting a new potential for the contrastive learning. We also investigate when such a latent space can be identified up to mixings within each component.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

25 of 35 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  5. 2018Group Normalizationcited by 1,514
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  8. 2019Contrastive Multiview Codingcited by 1,297
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2016Colorful Image Colorizationcited by 1,113
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
13 more
  1. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  2. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  3. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  4. 2021Towards Open World Object Detectioncited by 297
  5. 2018cGANs with Projection Discriminatorcited by 253
  6. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  7. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  8. 2021Automatic Symmetry Discovery with Lie Algebra Convolutional Networkcited by 82
  9. 2021Equivariant Contrastive Learningcited by 65
  10. 2021Self-Supervised Learning Disentangled Group Representation as Featurecited by 47
  11. 2014Learning the Irreducible Representations of Commutative Lie Groupscited by 41
  12. 2021Learning disentangled representations via product manifold projectioncited by 20
  13. 2021Addressing the Topological Defects of Disentanglement via Distributed Operatorscited by 14

Built on by

Not cited yet.