# Recurrent In-Context Learning for Tabular Foundation Models

arXiv: https://arxiv.org/abs/2610.14998 · HTML: https://mlsanity.com/paper/2610.14998/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Builds on (most cited first, 29 of 29)

- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Reasoning with Latent Thoughts: On the Power of Looped Transformers](https://mlsanity.com/paper/2502.17416/) (2025)
- [Looped Transformers as Programmable Computers](https://mlsanity.com/paper/2301.13196/) (2023)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Looped Transformers are Better at Learning Learning Algorithms](https://mlsanity.com/paper/2311.12424/) (2023)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation](https://mlsanity.com/paper/2507.10524/) (2025)
- [Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks](https://mlsanity.com/paper/2106.04537/) (2021)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [Looped Transformers for Length Generalization](https://mlsanity.com/paper/2409.15647/) (2024)
- [LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence](https://mlsanity.com/paper/2509.03505/) (2025)
- [Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA](https://mlsanity.com/paper/2410.20672/) (2024)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [A Mechanistic Analysis of Looped Reasoning Language Models](https://mlsanity.com/paper/2604.11791/) (2026)
- [Early Stopping Tabular In-Context Learning](https://mlsanity.com/paper/2506.21387/) (2025)
- [Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models](https://mlsanity.com/paper/2605.06510/) (2026)
- [TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention](https://mlsanity.com/paper/2606.07345/) (2026)
- [TabFM: A Zero-Shot Foundation Model for Tabular Data](https://mlsanity.com/paper/2609.37959/) (2026)
- [TACTICL: Task-Aware Compression of Tabular ICL Models](https://mlsanity.com/paper/2608.10837/) (2026)
- [Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation](https://mlsanity.com/paper/2608.15062/) (2026)

## Abstract (arXiv)

Tabular foundation models adapt to new datasets through in-context learning (ICL), and, like most transformers, they tie computational depth to the number of independently parameterized layers. We ask whether the ICL stage of such a model needs a distinct set of parameters for every layer, or only depth. We introduce Tab-Loop, which replaces the 12 untied ICL blocks of TabICLv2 with 4 tied blocks applied k times and changes nothing else: the encoders, decoder, synthetic prior and training recipe are shared with an untied baseline trained for the same compute. At matched pretraining compute (18.2 EFLOP) and matched inference compute (12 block applications), Tab-Loop matches the baseline with 62% fewer parameters (10.4M vs. 27.6M): held-out log loss is 0.4583 vs. 0.4588, mean accuracy on 200 TALENT datasets is 84.10% vs. 83.92%, and Elo on 38 TabArena datasets is 1409 vs. 1396. A 4-block untied control with Tab-Loop's parameter count, which sees 32% more training tables for the same compute, trails both (83.46%; Elo 1343). Because weights are shared, depth becomes a choice made at inference. Accuracy rises with the number of loops through the trained range and holds beyond it, and continued training with deeper loops extends the useful range, reaching the baseline's stage-2 scores at 4--5× its inference compute. At this scale, the ICL stage of a tabular foundation model needs depth more than it needs distinct parameters.