mlsanityData to 7 Oct 2026

A JEPA Recipe for Tabular Foundation Models

arXiv 2609.25541 ↗First posted 22 Sep 2026cs.LG cs.AIcited by 0

On the map: World models → Latent predictive world models

Abstract (arXiv)

Tabular foundation models learn to predict cell values in context, whereas world-model self-supervision asks for prediction in representation space (LeCun, 2022; Assran et al., 2023). On a tabular foundation-model prior, the latent term of a joint-embedding predictive architecture (JEPA) collapsed in our earlier runs and took the encoder with it to a constant map. We report a recipe under which the latent term survives to convergence beside the value objective: the value head reads the encoder field rather than the predictor, and the target is an exponential moving average (EMA) difference. To bound its cost against the value-only arm, both arms train until a plateau rule stops them, with no fixed step budget. A fixed horizon had confounded a slowdown with a ceiling, since the value-only arm was still improving well past the usual budget. At convergence, in one run per arm, the JEPA arm trails the value-only arm across 147 real datasets, 32:70 wins to losses on classification (29:63 with one entry per dataset name) and 8:24 on regression, the margin small on classification and wider on regression, and the count leans the same way in each stratum and each benchmark. The JEPA arm (jepa) needs 1.42 times as many steps as the value-only arm (ds), and 1.66 times its wall-clock, to reach its plateau.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 28 of its 33 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

28 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Revisiting Feature Prediction for Learning Visual Representations from Video15 Feb
  2. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture19 Jan

Other references · 26

  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  6. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  7. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  8. 2021Transformers Can Do Bayesian Inferencecited by 338
  9. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  10. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  11. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  12. 2017OpenML Benchmarking Suitescited by 169
14 more
  1. 2025TabArena: A Living Benchmark for Machine Learning on Tabular Datacited by 152
  2. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  3. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  4. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  5. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  6. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 101
  7. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  8. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 69
  9. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 67
  10. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 48
  11. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29
  12. 2023LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architecturescited by 27
  13. 2024T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Datacited by 13
  14. 2026Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Qualitycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Tabular Foundation Model for Generative Modelling
  2. 2026Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models
  3. 2026Distillation of Tabular Foundation Models into Efficient Predictors
  4. 2026TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models
  5. 2026TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention
  6. 2026Statistically Indistinguishable, Operationally Distinct: A Formal Barrier for Tabular Foundation Models
  7. 2025On Finetuning Tabular Foundation Models
  8. 2026Xiaomi-TabLDM: A Tabular Foundation Model Technical Report