mlsanityData to 14 Oct 2026

Elucidating the Role of Feature Normalization in IJEPA

arXiv 2508.02829 ↗First posted 4 Aug 2025cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

In the standard image joint embedding predictive architecture (IJEPA), features at the output of the teacher encoder are layer normalized (LN) before serving as a distillation target for the student encoder and predictor. We propose that this feature normalization disrupts the natural energy hierarchy of visual tokens, where high-energy tokens (those with larger L2 norms) encode semantically important image regions. LN forces all features to have identical L2 norms, effectively equalizing their energies and preventing the model from prioritizing semantically rich regions. We find that IJEPA models trained with feature LN exhibit loss maps with significant checkerboard-like artifacts. We propose that feature LN be replaced with a DynTanh activation as the latter better preserves token energies and allows high-energy tokens to greater contribute to the prediction loss. We show that IJEPA trained with feature DynTanh exhibits a longer-tailed loss distribution and fixes the checkerboard artifacts in the loss map. Our empirical results show that our simple modification improves ImageNet linear probe accuracy from 38% to 42.7% for ViT-Small and reduces RMSE by 0.08 on NYU Depth V2 monocular depth estimation. These results suggest that preserving natural token energies is crucial for effective self-supervised visual representation learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 13 of its 16 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

13 of 16 references.

References · 13

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2021Vision Transformers for Dense Predictioncited by 1,803
  6. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  7. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  8. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  9. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  10. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  11. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 91
  12. 2023LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architecturescited by 27
1 more
  1. 2025DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformerscited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
  2. 2026The JEPA Predictor: A Transferable Operator for Occluded Feature Completion
  3. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
  4. 2026MotionJEPA: Preventing Temporal Feature Collapse by Capturing Visual Changes in Latent Space
  5. 2024Learning and Leveraging World Models in Visual Representation Learning
  6. 2026Text-Conditional JEPA for Learning Semantically Rich Visual Representations
  7. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
  8. 2026UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling