# Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA

arXiv: https://arxiv.org/abs/2602.22617 · HTML: https://mlsanity.com/paper/2602.22617/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 54 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 30 of 37)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Infinite attention: NNGP and NTK for deep attention networks](https://mlsanity.com/paper/2006.10540/) (2020)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?](https://mlsanity.com/paper/1511.05101/) (2015)
- [SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates](https://mlsanity.com/paper/2008.10546/) (2020)
- [OpenELM: An Efficient Language Model Family with Open Training and Inference Framework](https://mlsanity.com/paper/2404.14619/) (2024)
- [Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics](https://mlsanity.com/paper/2105.03703/) (2021)
- [LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures](https://mlsanity.com/paper/2509.14252/) (2025)
- [Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language](https://mlsanity.com/paper/2311.04930/) (2023)
- [How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks](https://mlsanity.com/paper/2407.03475/) (2024)
- [Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization](https://mlsanity.com/paper/2202.00553/) (2022)
- [Statistical exploration of the Manifold Hypothesis](https://mlsanity.com/paper/2208.11665/) (2022)

## Built on by (most cited first, 7 of 7)

- [The Topological Trouble With Transformers](https://mlsanity.com/paper/2604.17121/) (2026)
- [SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models](https://mlsanity.com/paper/2606.32022/) (2026)
- [Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling](https://mlsanity.com/paper/2604.18464/) (2026)
- [Consistent Geometric Deep Learning via Hilbert Bundles and Cellular Sheaves](https://mlsanity.com/paper/2605.06395/) (2026)
- [Representation Without Reward: A JEPA Audit for LLM Fine-Tuning](https://mlsanity.com/paper/2605.15394/) (2026)
- [How Psychological Learning Paradigms Shaped and Constrained Artificial Intelligence](https://mlsanity.com/paper/2603.18203/) (2026)
- [Representational Curvature Modulates Behavioral Uncertainty in Large Language Models](https://mlsanity.com/paper/2604.23985/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) obey consistent scaling laws -- empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these laws are descriptive rather than prescriptive: they characterize typical training, not optimal training. Surprisingly few works have successfully challenged the data-efficiency bounds implied by these laws -- which is our primary focus. To that end, we introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear. Building on this principle, we propose a novel Semantic Tube Prediction (STP) task, a JEPA-style regularizer that confines hidden-state trajectories to a tubular neighborhood of the geodesic. STP generalizes JEPA to language without requiring explicit multi-view augmentations. We show this constraint improves signal-to-noise ratio, and consequently preserves diversity by preventing trajectory collisions during inference. Empirically, STP allows LLMs to match baseline accuracy with 16× less training data on the NL-RX-SYNTH dataset, directly violating the data term of Chinchilla-style scaling laws and demonstrating that principled geometric priors can surpass brute-force scaling. Code is available at https://github.com/galilai-group/llm-jepa#stp.