mlsanityData to 7 Oct 2026

PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment

arXiv 2605.17671 ↗First posted 17 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Non-contrastive self-supervised learning (SSL) is an effective framework for predictive representation learning, but popular (and in practice effective) methods such as SimSiam, BYOL, I-JEPA or DINO, which rely on a form of self-distillation to train a teacher-student network, remain poorly understood as they typically do not minimize a well-defined objective. We analyze the dynamics of a variant of the Joint Embedding Predictive Architecture (JEPA) using a regularized linear regressor to predict the learned representations of two views of the data from one another, and fully characterize its stability: non-collapsed stable equilibria align with leading nonlinear canonical correlation subspaces, while collapsed equilibria may also be stable attractors. Motivated by this result, we introduce PEIRA, a non-contrastive SSL method with an explicit objective defined through the trace of the optimal linear regressor. We show that its only stable equilibria are nontrivial global minimizers and recover the same canonical correlation subspaces, with regularization selecting the effective dimension. Experiments on ImageNet-1K and CIFAR-10 show PEIRA is competitive with VICReg and LeJEPA baselines, and qualitative empirical results support the theory.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 72 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

42 of 72 references.

References · 42

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  9. 2025DINOv3cited by 1,297
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
30 more
  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  3. 2017Large Batch Training of Convolutional Networkscited by 529
  4. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  5. 2024The Platonic Representation Hypothesiscited by 362
  6. 2020Whitening for Self-Supervised Representation Learningcited by 278
  7. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  8. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  9. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  10. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 138
  11. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  12. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  13. 2014Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functionscited by 93
  14. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
  15. 2020Learning Deep Features in Instrumental Variable Regressioncited by 65
  16. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 65
  17. 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 62
  18. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 40
  19. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  20. 2021Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspectivecited by 37
  21. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 37
  22. 2020On Learning Rates and Schr"odinger Operatorscited by 33
  23. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29
  24. 2014Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problemscited by 25
  25. 2015Nonparametric Canonical Correlation Analysiscited by 22
  26. 2026A Lightweight Library for Energy-Based Joint-Embedding Predictive Architecturescited by 13
  27. 2023Unconstrained Stochastic CCA: Unifying Multiview and Self-Supervised Learningcited by 6
  28. 2020Some convergent results for Backtracking Gradient Descent method on Banach spacescited by 3
  29. 2025Dual Perspectives on Non-Contrastive Self-Supervised Learningcited by 3
  30. 2025Closed-Form Last Layer Optimizationcited by 3

Built on by

Not cited yet.