# TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity

arXiv: https://arxiv.org/abs/2610.07559 · HTML: https://mlsanity.com/paper/2610.07559/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 60 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 14 of 14)

- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence](https://mlsanity.com/paper/2509.03505/) (2025)
- [Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data](https://mlsanity.com/paper/2507.03971/) (2025)
- [Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers](https://mlsanity.com/paper/2405.13396/) (2024)
- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)
- [Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models](https://mlsanity.com/paper/2605.06510/) (2026)
- [LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models](https://mlsanity.com/paper/2606.04485/) (2026)
- [Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality](https://mlsanity.com/paper/2605.18971/) (2026)
- [Interpretable Tabular Foundation Models via In-Context Kernel Regression](https://mlsanity.com/paper/2602.02162/) (2026)

## Abstract (arXiv)

Recent progress in tabular foundation models suggests that training on synthetic tasks can substantially improve in-context learning capabilities, with overall performance largely depending on how well models can infer task-specific predictive relationships from the available context during inference. In this paper, we introduce TAFFY, a tabular foundation model with an In-Context Diversity Prior and a Task-Conditioned Looped Transformer that strengthen this ability. Specifically, to construct each synthetic pretraining context, the In-Context Diversity Prior samples from multiple related environments derived via controlled interventions and distribution shifts on a shared causal process. This in-context diversity encourages the model to learn a more comprehensive and task-specific representation. Moreover, the Task-Conditioned Looped Transformer iteratively and selectively applies a shared group of Transformer blocks to refine contextual representations, with a task-conditioned gate modulating the final hidden-state update. This enables task-adaptive iterative refinement. Together, these components encourage the model to identify predictive relationships from contextual contrasts during pretraining and dynamically modulate context integration for each task. Across six classification and five regression benchmark datasets, TAFFY attains the lowest average rank.