mlsanityData to 7 Oct 2026

MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features

arXiv 2307.12698 ↗First posted 24 Jul 2023cs.CV cs.AI cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning of visual representations has been focusing on learning content features, which do not capture object motion or location, and focus on identifying and differentiating objects in images and videos. On the other hand, optical flow estimation is a task that does not involve understanding the content of the images on which it is estimated. We unify the two approaches and introduce MC-JEPA, a joint-embedding predictive architecture and self-supervised learning approach to jointly learn optical flow and content features within a shared encoder, demonstrating that the two associated objectives; the optical flow estimation objective and the self-supervised learning objective; benefit from each other and thus learn content features that incorporate motion information. The proposed approach achieves performance on-par with existing unsupervised optical flow benchmarks, as well as with common self-supervised learning approaches on downstream tasks such as semantic segmentation of images and videos.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 68 of its 87 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

68 of 87 references.

References · 68

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2016Layer Normalizationcited by 5,579
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
38 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  7. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  8. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  9. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  10. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  11. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  12. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  13. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  14. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  15. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  16. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  17. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  18. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  19. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  20. 2020Whitening for Self-Supervised Representation Learningcited by 278
  21. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  22. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  23. 2021Learning to Estimate Hidden Motions with Global Motion Aggregationcited by 223
  24. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  25. 2017Multi-task Self-Supervised Visual Learningcited by 200
  26. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  27. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  28. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  29. 2020Anomaly Detection in Video via Self-Supervised and Multi-Task Learningcited by 147
  30. 2019SelFlow: Self-Supervised Learning of Optical Flowcited by 128
  31. 2020What Matters in Unsupervised Optical Flowcited by 124
  32. 2020Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimationcited by 120
  33. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  34. 2020MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Maskcited by 117
  35. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  36. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  37. 2016Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothnesscited by 101
  38. 2021Instance Localization for Self-supervised Detection Pretrainingcited by 99

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 14; 2025: 12; 2026: 23.

53 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  2. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  3. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  4. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  5. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  6. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  7. 2025Embodied AI: From LLMs to World Modelscited by 23
  8. 2023Graph-level Representation Learning with Joint-Embedding Predictive Architecturescited by 21
  9. 2024SIGMA: Sinkhorn-Guided Masked Video Modelingcited by 14
  10. 20243D-JEPA: A Joint Embedding Predictive Architecture for 3D Self-Supervised Representation Learningcited by 14
  11. 2026VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Modelscited by 14
  12. 2024M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecturecited by 12
38 more
  1. 2024Denoising with a Joint-Embedding Predictive Architecturecited by 12
  2. 2024T-JEPA: A Joint-Embedding Predictive Architecture for Trajectory Similarity Computationcited by 11
  3. 2024PooDLe: Pooled and dense self-supervised learning from naturalistic videoscited by 11
  4. 2026World Action Models: A Surveycited by 10
  5. 2025Recurrent Video Masked Autoencoderscited by 9
  6. 2026SkyJEPA: Learning Long-Horizon World Models for Zero-Shot Sim-to-Real Control of Quadrotorscited by 9
  7. 2023Guarding Barlow Twins Against Overfitting with Mixed Samplescited by 7
  8. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  9. 2025EgoAgent: A Joint Predictive Agent Model in Egocentric Worldscited by 7
  10. 2024LaT-PFN: A Joint Embedding Predictive Architecture for In-context Time-series Forecastingcited by 6
  11. 2026Orca: The World is in Your Mindcited by 6
  12. 2024VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Trainingcited by 5
  13. 2026You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differencescited by 5
  14. 2025Self-Supervised Learning of Motion Concepts by Optimizing Counterfactualscited by 4
  15. 2026ContactWorld: What Representations Matter for Vision-Tactile Latent World Models in Contact-Rich Manipulationcited by 3
  16. 2025Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptionscited by 2
  17. 2025Discrete JEPA: Learning Discrete Token Representations without Reconstructioncited by 2
  18. 2025Midway Network: Learning Representations for Recognition and Motion from Latent Dynamicscited by 2
  19. 2026Capturing Visual Environment Structure Correlates with Control Performancecited by 2
  20. 2026Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddingscited by 2
  21. 2025A New Perspective On AI Safety Through Control Theory Methodologiescited by 1
  22. 2025Unifying Vision-Language Latents for Zero-label Image Caption Enhancementcited by 1
  23. 2026JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamicscited by 1
  24. 2024Joint-Embedding Predictive Architecture for Self-Supervised Learning of Mask Classification Architecturecited by 0
  25. 2024Predictive Attractor Modelscited by 0
  26. 2025SparseJEPA: Sparse Representation Learning of Joint Embedding Predictive Architecturescited by 0
  27. 2025Optimisation Is Not What You Needcited by 0
  28. 2025Generalized Event Partonomy Inference with Structured Hierarchical Predictive Learningcited by 0
  29. 2026Online Monitoring Framework for Automotive Time Series Data using JEPA Embeddingscited by 0
  30. 2026FluidWorld: Reaction-Diffusion Dynamics as a Predictive Substrate for World Modelscited by 0
  31. 2026UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architecturescited by 0
  32. 2026Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learningcited by 0
  33. 2026SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictorscited by 0
  34. 2026Toward a Causal Data Management Ecosystem for Decision Making and Agentic AIcited by 0
  35. 2026Predicting Functions, Not Features: KANs with Function-Space Joint-Embedding Predictive Learning for Medical Image Segmentationcited by 0
  36. 2026Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipatescited by 0
  37. 2026Representation Learning in Diffusion and Flow-based Model: An Application Aspectcited by 0
  38. 2026Asymmetric Within-Document Predictive Learning for Scientific Document Representationcited by 0

50 most cited of 53 All 53 (JSON)