# LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures

arXiv: https://arxiv.org/abs/2509.14252 · HTML: https://mlsanity.com/paper/2509.14252/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 50 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)

## Built on by (most cited first, 30 of 38)

- [Next-Latent Prediction Transformers Learn Compact World Models](https://mlsanity.com/paper/2511.05963/) (2025)
- [JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures](https://mlsanity.com/paper/2602.17162/) (2026)
- [Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA](https://mlsanity.com/paper/2602.22617/) (2026)
- [Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models](https://mlsanity.com/paper/2512.15885/) (2025)
- [Multimodal Latent Reasoning via Predictive Embeddings](https://mlsanity.com/paper/2604.08065/) (2026)
- [Crys-JEPA: Accelerating Crystal Discovery via Embedding Screening and Generative Refinement](https://mlsanity.com/paper/2605.14759/) (2026)
- [The Patient is not a Moving Document: A World Model Training Paradigm for Longitudinal EHR](https://mlsanity.com/paper/2601.22128/) (2026)
- [Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression](https://mlsanity.com/paper/2603.23308/) (2026)
- [Pantagruel: Unified Self-Supervised Encoders for French Text and Speech](https://mlsanity.com/paper/2601.05911/) (2026)
- [LLM2Vec-Gen: Generative Embeddings from Large Language Models](https://mlsanity.com/paper/2603.10913/) (2026)
- [Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectories](https://mlsanity.com/paper/2605.10840/) (2026)
- [What Does a Chemical Language Model Know About Molecules?](https://mlsanity.com/paper/2606.23443/) (2026)
- [Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models](https://mlsanity.com/paper/2607.12962/) (2026)
- [The JEPA Paradox in Language: The Geometry of Linguistic Alternatives](https://mlsanity.com/paper/2607.23531/) (2026)
- [Alternatives To Next Token Prediction In Text Generation -- A Survey](https://mlsanity.com/paper/2509.24435/) (2025)
- [BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics](https://mlsanity.com/paper/2601.00366/) (2026)
- [The Mouth is Not the Brain: Bridging Energy-Based World Models and Language Generation](https://mlsanity.com/paper/2601.17094/) (2026)
- [EIDOS: Latent-Space Predictive Learning for Time Series Foundation Models](https://mlsanity.com/paper/2602.14024/) (2026)
- [Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models](https://mlsanity.com/paper/2603.29123/) (2026)
- [Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling](https://mlsanity.com/paper/2604.18464/) (2026)
- [Representation Without Reward: A JEPA Audit for LLM Fine-Tuning](https://mlsanity.com/paper/2605.15394/) (2026)
- [Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language](https://mlsanity.com/paper/2605.15607/) (2026)
- [DLLM-JEPA: Joint Embedding Predictive Architectures for Masked Diffusion Language Models](https://mlsanity.com/paper/2606.00091/) (2026)
- [Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning](https://mlsanity.com/paper/2606.05173/) (2026)
- [Do Reasoning Models Enhance Embedding Models?](https://mlsanity.com/paper/2601.21192/) (2026)
- [ProteinJEPA: Latent prediction improves protein language model pretraining](https://mlsanity.com/paper/2605.07554/) (2026)
- [Beyond Thinking: Imagining in 360^circ for Humanoid Visual Search](https://mlsanity.com/paper/2605.09146/) (2026)
- [GRAG: Generic Response-Augmented Generation Framework for Personalized Conversational Systems](https://mlsanity.com/paper/2606.21097/) (2026)
- [Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization](https://mlsanity.com/paper/2607.17019/) (2026)
- [BayesPrompt: human readable prompts that make sense](https://mlsanity.com/paper/2608.17866/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) pretraining, finetuning, and evaluation rely on input-space reconstruction and generative capabilities. Yet, it has been observed in vision that embedding-space training objectives, e.g., with Joint Embedding Predictive Architectures (JEPAs), are far superior to their input-space counterpart. That mismatch in how training is achieved between language and vision opens up a natural question: em can language training methods learn a few tricks from the vision ones? The lack of JEPA-style LLM is a testimony of the challenge in designing such objectives for language. In this work, we propose a first step in that direction where we develop LLM-JEPA, a JEPA based solution for LLMs applicable both to finetuning and pretraining. Thus far, LLM-JEPA is able to outperform the standard LLM training objectives by a significant margin across models, all while being robust to overfiting. Those findings are observed across numerous datasets (NL-RX, GSM8K, Spider, RottenTomatoes) and various models from the Llama3, OpenELM, Gemma2 and Olmo families. Code: https://github.com/rbalestr-lab/llm-jepa.