Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
Not on the map. Why?
Abstract (arXiv)
Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure. Inspired by the success of Joint Embedding Predictive Architectures (JEPA) (LeCun, 2022) in vision and audio, we propose a hybrid pre-training objective that combines a JEPA-style latent-space prediction loss with a standard MLM objective over a single shared encoder. A learnable scalar parameter continuously balances the two objectives during training. We pre-train both a hybrid model and a pure-MLM baseline on English Wikipedia using identical architectures and compute budgets (NVIDIA H100). Extensive representation analysis across five GLUE benchmarks (SST-2, MRPC, MNLI, CoLA, STS-B) using four pooling strategies reveals that the hybrid encoder produces significantly more uniform embeddings (uniformity less than -0.16 vs -0.05 for MLM), exhibits richer spectral geometry under max pooling, encodes less surface-level lexical information, and achieves a better semantic-to-lexical balance. Despite similar linear-probe downstream accuracy, the geometric differences are consistent and significant, suggesting that the JEPA predictive objective reshapes the latent space in ways that standard accuracy metrics alone cannot capture.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 16 Apr 2026.
Builds on
23 of 28 references.
References · 23
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
11 more
- 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
- 2020A Primer in BERTology: What we know about how BERT workscited by 815
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
- 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
- 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
- 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
- 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
- 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
- 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 57
- 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 38
Built on by
Citing papers per year: 2026: 1.
1 paper.