mlsanityData to 14 Oct 2026

Video Representation Learning with Joint-Embedding Predictive Architectures

arXiv 2412.10925 ↗First posted 14 Dec 2024cs.CV cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Video representation learning is an increasingly important topic in machine learning research. We present Video JEPA with Variance-Covariance Regularization (VJ-VCR): a joint-embedding predictive architecture for self-supervised video representation learning that employs variance and covariance regularization to avoid representation collapse. We show that hidden representations from our VJ-VCR contain abstract, high-level information about the input data. Specifically, they outperform representations obtained from a generative baseline on downstream tasks that require understanding of the underlying dynamics of moving objects in the videos. Additionally, we explore different ways to incorporate latent variables into the VJ-VCR framework that capture information about uncertainty in the future in non-deterministic settings.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

32 of 42 references.

References · 32

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2018SlowFast Networks for Video Recognitioncited by 1,867
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2021ViViT: A Video Vision Transformercited by 1,620
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  12. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
20 more
  1. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  2. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  3. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  4. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  5. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  6. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  7. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  8. 2018Stochastic Video Generation with a Learned Priorcited by 245
  9. 2022SimVP: Simpler yet Better Video Predictioncited by 184
  10. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  11. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  12. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  13. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  14. 2023OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learningcited by 54
  15. 2022Understanding Collapse in Non-Contrastive Siamese Representation Learningcited by 42
  16. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  17. 2022A Comprehensive Review on Autonomous Navigationcited by 19
  18. 2023Variance-Covariance Regularization Improves Representation Learningcited by 14
  19. 2017Prediction Under Uncertainty with Error-Encoding Networkscited by 6
  20. 2019Evolving Losses for Unlabeled Video Representation Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2026Learning Latent Action World Models In The Wildcited by 47
  2. 2025Closing the Train-Test Gap in World Models for Gradient-Based Planningcited by 11
  3. 2025Why and How Auxiliary Tasks Improve JEPA Representationscited by 5
  4. 2026Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddingscited by 5
  5. 2026Var-JEPA: A Variational Formulation of the Joint-Embedding Predictive Architecture - Bridging Predictive and Generative Self-Supervised Learningcited by 4
  6. 2026AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modelingcited by 4
  7. 2025Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptionscited by 2
  8. 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoningcited by 2
  9. 2026Self-Supervised Learning from Structural Invariancecited by 1
  10. 2026Masked and Predictive Self-Supervised Foundation Models for 3D Brain MRIcited by 1
  11. 2025Meta-Representational Predictive Coding: Neuroscience-Informed Self-Supervised Learningcited by 0
  12. 2025SpikeGen: Decoupled "Rods and Cones" Visual Representation Processing with Latent Generative Frameworkcited by 0
3 more
  1. 2026Normative Networks for Source Separation via Local Plasticity and Dendritic Computationcited by 0
  2. 2026Neural Voxel Dynamics: Learning Volumetric Feature Advection for 3D Physics in V-JEPA Latent Spacecited by 0
  3. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0