mlsanityData to 14 Oct 2026

ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models

arXiv 2609.36952 ↗First posted 29 Sep 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) excel at token-level generation but may learn undesirable abstract semantics and lack comprehensive perception. LLM-JEPA mitigates this by aligning different views of the same underlying knowledge via a joint-embedding predictive architecture (JEPA). However, strong alignment does not necessarily lead to accurate, stable predictions. To address this, we propose ER-JEPA, which adds an episodic replay path to LLM-JEPA. ER-JEPA stores training pairs in a memory. At each step, it stores and retrieves relevant data to provide additional supervision. This enables learning from both the current batch and stored training pairs, providing additional supervision for token prediction and representation alignment. Experiments across multiple datasets (NL-RX, GSM8K, Spider, and NQ-Open) demonstrate that ER-JEPA consistently outperforms LLM-JEPA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

26 of 30 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  7. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  8. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  10. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  11. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  12. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
14 more
  1. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  3. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  4. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
  5. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  6. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  7. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 72
  8. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  9. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 38
  10. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 31
  11. 2023Emergent Representations of Program Semantics in Language Models Trained on Programscited by 30
  12. 2024Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Taskscited by 25
  13. 2016Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledgecited by 24
  14. 2025Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspectivecited by 10

Built on by

Not cited yet.