mlsanityData to 14 Oct 2026

Representation Without Reward: A JEPA Audit for LLM Fine-Tuning

arXiv 2605.15394 ↗First posted 14 May 2026cs.LG cs.AI stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

Joint-embedding predictive architectures (JEPAs) propose that a model should learn more useful abstractions when trained to predict latent representations rather than observed outputs. For autoregressive language-model fine-tuning the principle entails a stricter requirement: the induced hidden-state geometry must reach the language-model head and improve the decoded task metric. We test that requirement under a fixed Llama-3.2-1B-Instruct LoRA harness on natural-language-to-regex generation, comparing twenty-two training-time auxiliaries across trajectory-shape regularisation, distributional constraints, predictor/target asymmetry, Fisher-metric Jacobi residuals, and a decoder-visible JEPA objective constructed to lie in cross-entropy's positive cone. The empirical answer is a structured null: several auxiliaries clear single-cell paired α = 0.10 without correction (T3-Local at Delta = +2.53~pp, p = 0.003 being the strongest), but none survives Bonferroni or Holm--Bonferroni at the relevant family-wise threshold, even though many change curvature, anisotropy, variance, and gradient direction. Decoder-visible JEPA yields the first positive auxiliary--cross-entropy gradient cosine in the study, yet exact match remains inside seed noise; a full-fine-tuning replication of the same auxiliary at n = 5 seeds reproduces the null on both benchmarks (TURK: Delta = +0.04~pp, ppaired = 0.96; SYNTH: Delta = +0.52~pp, ppaired = 0.28), so the null is robust across LoRA and full fine-tuning for the decoder-visible construction. Hidden-state representation work and decoded-task accuracy are therefore weakly coupled in this regime; we accordingly reframe LLM-domain JEPA evaluation as a coupling problem, in which the operative question is under which metrics useful hidden geometry becomes decoder-visible task signal.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

24 of 34 references.

References · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  8. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  9. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  12. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
12 more
  1. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  2. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  3. 2019Generalized Sliced Wasserstein Distancescited by 208
  4. 2024The pitfalls of next-token predictioncited by 160
  5. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  6. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  7. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 38
  8. 2026Temporal Straightening for Latent Planningcited by 38
  9. 2023Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural languagecited by 32
  10. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 31
  11. 2022Implicit variance regularization in non-contrastive SSLcited by 16
  12. 2026Semantic Tube Prediction: Beating LLM Data Efficiency with JEPAcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantizationcited by 0