mlsanityData to 7 Oct 2026

BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics

arXiv 2601.00366 ↗First posted 1 Jan 2026cs.CL cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a training paradigm that adds a JEPA training objective to BERT-style models, working to combat a collapsed [CLS] embedding space and turning it into a language-agnostic space. This new structure leads to increased performance across multilingual benchmarks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 1 Jan 2026.

Builds on

22 of 40 references.

References · 22

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
10 more
  1. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  2. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  4. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  5. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  6. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  7. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 61
  8. 2021Theoretical Foundations of t-SNE for Visualizing High-Dimensional Clustered Datacited by 55
  9. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 35
  10. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Bongard: Training Machine Intuitioncited by 0