mlsanityData to 14 Oct 2026

LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures

arXiv 2509.14252 ↗First posted 11 Sep 2025cs.CL cs.AIcited by 38

Not on the map. Why?

Abstract (arXiv)

Large Language Model (LLM) pretraining, finetuning, and evaluation rely on input-space reconstruction and generative capabilities. Yet, it has been observed in vision that embedding-space training objectives, e.g., with Joint Embedding Predictive Architectures (JEPAs), are far superior to their input-space counterpart. That mismatch in how training is achieved between language and vision opens up a natural question: em can language training methods learn a few tricks from the vision ones? The lack of JEPA-style LLM is a testimony of the challenge in designing such objectives for language. In this work, we propose a first step in that direction where we develop LLM-JEPA, a JEPA based solution for LLMs applicable both to finetuning and pretraining. Thus far, LLM-JEPA is able to outperform the standard LLM training objectives by a significant margin across models, all while being robust to overfiting. Those findings are observed across numerous datasets (NL-RX, GSM8K, Spider, RottenTomatoes) and various models from the Llama3, OpenELM, Gemma2 and Olmo families. Code: https://github.com/rbalestr-lab/llm-jepa.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 47 of its 50 references that are in the map. First posted on arXiv 11 Sep 2025.

Builds on

47 of 50 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
35 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  9. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  10. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  13. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  14. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  15. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  16. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  17. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  18. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  19. 20242 OLMo 2 Furiouscited by 340
  20. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  21. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  22. 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 195
  23. 2017Learning a Neural Semantic Parser from User Feedbackcited by 99
  24. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  25. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 75
  26. 2024Learning by Reconstruction Produces Uninformative Features For Perceptioncited by 55
  27. 2020CC2Vec: Distributed Representations of Code Changescited by 39
  28. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 31
  29. 2016Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledgecited by 24
  30. 2020Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repaircited by 22
  31. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13
  32. 2019Commit2Vec: Learning Distributed Representations of Code Changescited by 12
  33. 2019Sketch-Driven Regular Expression Generation from Natural Language and Examplescited by 10
  34. 2023CCBERT: Self-Supervised Code Change Representation Learningcited by 6
  35. 2025JEPA for RL: Investigating Joint-Embedding Predictive Architectures for Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 35.

38 papers.

Sort:

By area:

  1. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 31
  2. 2026JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecturescited by 9
  3. 2026Semantic Tube Prediction: Beating LLM Data Efficiency with JEPAcited by 7
  4. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  5. 2026Multimodal Latent Reasoning via Predictive Embeddingscited by 4
  6. 2026Crys-JEPA: Accelerating Crystal Discovery via Embedding Screening and Generative Refinementcited by 4
  7. 2026The Patient is not a Moving Document: A World Model Training Paradigm for Longitudinal EHRcited by 3
  8. 2026Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compressioncited by 3
  9. 2026Pantagruel: Unified Self-Supervised Encoders for French Text and Speechcited by 2
  10. 2026LLM2Vec-Gen: Generative Embeddings from Large Language Modelscited by 2
  11. 2026Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectoriescited by 2
  12. 2026What Does a Chemical Language Model Know About Molecules?cited by 2
26 more
  1. 2026Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Modelscited by 2
  2. 2026The JEPA Paradox in Language: The Geometry of Linguistic Alternativescited by 2
  3. 2025Alternatives To Next Token Prediction In Text Generation -- A Surveycited by 1
  4. 2026BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semanticscited by 1
  5. 2026The Mouth is Not the Brain: Bridging Energy-Based World Models and Language Generationcited by 1
  6. 2026EIDOS: Latent-Space Predictive Learning for Time Series Foundation Modelscited by 1
  7. 2026Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Modelscited by 1
  8. 2026Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Samplingcited by 1
  9. 2026Representation Without Reward: A JEPA Audit for LLM Fine-Tuningcited by 1
  10. 2026Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Languagecited by 1
  11. 2026DLLM-JEPA: Joint Embedding Predictive Architectures for Masked Diffusion Language Modelscited by 1
  12. 2026Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learningcited by 1
  13. 2026Do Reasoning Models Enhance Embedding Models?cited by 0
  14. 2026ProteinJEPA: Latent prediction improves protein language model pretrainingcited by 0
  15. 2026Beyond Thinking: Imagining in 360^circ for Humanoid Visual Searchcited by 0
  16. 2026GRAG: Generic Response-Augmented Generation Framework for Personalized Conversational Systemscited by 0
  17. 2026Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantizationcited by 0
  18. 2026BayesPrompt: human readable prompts that make sensecited by 0
  19. 2026Asymmetric Within-Document Predictive Learning for Scientific Document Representationcited by 0
  20. 2026Distributed JEPA: A Self-Supervised Framework for Energy Forecastingcited by 0
  21. 2026One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusioncited by 0
  22. 2026ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Modelscited by 0
  23. 2026Bongard: Training Machine Intuitioncited by 0
  24. 2026Latent JEPA: Abstract Future Prediction for Latent Reasoning in Chemistrycited by 0
  25. 2026LS-AR: Future-Predictive Latent Steering in Autoregressive LLMscited by 0
  26. 2026Isotropic Yet Undecodable: The Sequential Content-Sufficiency Gap in Latent-Predictive Text Representationscited by 0