mlsanityData to 14 Oct 2026

Factorized Latent Dynamics for Video JEPA: An Empirical Study of Auxiliary Objectives

arXiv 2605.17165 ↗First posted 16 May 2026cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Joint-Embedding Predictive Architectures (JEPA) are a promising framework for self-supervised video representation learning, yet the behavior of auxiliary objectives in small-scale Video-JEPA training is not well characterized. We report a small-scale empirical study of 18 auxiliary objective variants for Video-JEPA across two pretraining regimes: single-dataset (UCF-101) and mixed-dataset (UCF-101 + Something-Something V2 + ImageNet-100). We evaluate frozen representations on three complementary benchmarks: Diving-48 (fine-grained motion), SomethingSomething V2 (temporal reasoning), and ImageNet-100 (appearance). Our experiments suggest that many auxiliary objectives exhibit capacity trade-offs: gains on one downstream capability often coincide with degradation on another. We then study FWM-HW-LD (Factorized World-Model with Hard-Region-Weighted Latent Dynamics), a training-time objective that separates the latent representation into appearance and dynamics subspaces and applies hard-region weighting to both JEPA prediction errors and latent dynamics errors. In our mixed-dataset setting, FWM-HW-LD improves ImageNet-100 by +5.92 and SSv2 by +3.21 percentage points relative to the reference baseline, while remaining within 0.30 percentage points on Diving-48. These results indicate that latent factorization is a useful direction for studying auxiliary-objective trade-offs in Video-JEPA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 61 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

40 of 61 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
28 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  2. 2018SlowFast Networks for Video Recognitioncited by 1,867
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  4. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  5. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  6. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  8. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  9. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  11. 2021Video Swin Transformercited by 1,091
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  13. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  14. 2021Multiscale Vision Transformerscited by 897
  15. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  16. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  17. 2020Rescaling Egocentric Visioncited by 540
  18. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  19. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  20. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  21. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  22. 2016Learning Features by Watching Objects Movecited by 138
  23. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  24. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  25. 2023Siamese Masked Autoencoderscited by 76
  26. 2020Cycle-Contrast for Self-Supervised Video Representation Learningcited by 34
  27. 2018Video Representation Learning Using Discriminative Poolingcited by 7
  28. 2025Why and How Auxiliary Tasks Improve JEPA Representationscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
  2. 2026A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
  3. 2026Entity-Centric World Models: Interaction-Aware Masking for Causal Video Prediction
  4. 2026MotionJEPA: Preventing Temporal Feature Collapse by Capturing Visual Changes in Latent Space
  5. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
  6. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
  7. 2026VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Models
  8. 2022Joint Embedding Predictive Architectures Focus on Slow Features