# ConTextTab: A Semantics-Aware Tabular In-Context Learner

arXiv: https://arxiv.org/abs/2506.10707 · HTML: https://mlsanity.com/paper/2506.10707/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Self-attention Does Not Need O(n2) Memory](https://mlsanity.com/paper/2112.05682/) (2021)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data](https://mlsanity.com/paper/2407.04491/) (2024)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [CARTE: Pretraining and Transfer for Tabular Learning](https://mlsanity.com/paper/2402.16785/) (2024)
- [Large Scale Transfer Learning for Tabular Data via Language Modeling](https://mlsanity.com/paper/2406.12031/) (2024)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [A Closer Look at Deep Learning Methods on Tabular Datasets](https://mlsanity.com/paper/2407.00956/) (2024)
- [TabLib: A Dataset of 627M Tables with Context](https://mlsanity.com/paper/2310.07875/) (2023)
- [Mambular: A Sequential Model for Tabular Deep Learning](https://mlsanity.com/paper/2408.06291/) (2024)
- [PORTAL: Scalable Tabular Foundation Models via Content-Specific Tokenization](https://mlsanity.com/paper/2410.13516/) (2024)
- [ClusterTabNet: Supervised clustering method for table detection and table structure recognition](https://mlsanity.com/paper/2402.07502/) (2024)

## Built on by (most cited first, 30 of 34)

- [Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2511.02818/) (2025)
- [PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models](https://mlsanity.com/paper/2602.04029/) (2026)
- [KumoRFM-2: Scaling Foundation Models for Relational Learning](https://mlsanity.com/paper/2604.12596/) (2026)
- [TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Models](https://mlsanity.com/paper/2511.02802/) (2025)
- [TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrieval](https://mlsanity.com/paper/2511.03570/) (2025)
- [End-to-End Compression for Tabular Foundation Models](https://mlsanity.com/paper/2602.05649/) (2026)
- [Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks](https://mlsanity.com/paper/2604.21696/) (2026)
- [Mitra-v2 Technical Report](https://mlsanity.com/paper/2609.04540/) (2026)
- [Comparing Task-Agnostic Embedding Models for Tabular Data](https://mlsanity.com/paper/2511.14276/) (2025)
- [Towards Pretraining Text Encoders for TabPFN](https://mlsanity.com/paper/2606.04876/) (2026)
- [LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains](https://mlsanity.com/paper/2511.06161/) (2025)
- [Language Model Representations for Efficient Few-Shot Tabular Classification](https://mlsanity.com/paper/2602.15844/) (2026)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks](https://mlsanity.com/paper/2606.02384/) (2026)
- [Understanding the Surprising Generalization Properties of Tabular Foundation Models](https://mlsanity.com/paper/2608.17957/) (2026)
- [RELATE: A Schema-Agnostic Perceiver Encoder for Multimodal Relational Graphs](https://mlsanity.com/paper/2510.19954/) (2025)
- [Tabular Foundation Models Can Learn Association Rules](https://mlsanity.com/paper/2602.14622/) (2026)
- [Exploring Differences Between Tabular Enterprise Data and Public Benchmarks](https://mlsanity.com/paper/2606.30452/) (2026)
- [Probing Memorization of Tabular In-Context Learning](https://mlsanity.com/paper/2606.31208/) (2026)
- [Enhancing Tabular Learners with Context-Aware Semantic Embeddings](https://mlsanity.com/paper/2608.03565/) (2026)
- [Causal Foundation Models](https://mlsanity.com/paper/2609.03003/) (2026)
- [SALT-KG: A Benchmark for Semantics-Aware Learning on Enterprise Tables](https://mlsanity.com/paper/2601.07638/) (2026)
- [TFM-Retouche: A Lightweight Input-Space Adapter for Tabular Foundation Models](https://mlsanity.com/paper/2605.06047/) (2026)
- [Data Language Models: A New Foundation Model Class for Tabular Data](https://mlsanity.com/paper/2605.06290/) (2026)
- [Are Tabular Foundation Models Robust to Realistic Query Distribution Shifts in Microbiome Data?](https://mlsanity.com/paper/2606.24995/) (2026)
- [FlexTab: A Flexible Encoder-Decoder Architecture for In-Context Learning Across Diverse Tabular Tasks](https://mlsanity.com/paper/2606.30336/) (2026)
- [Human Preference aligned Tabular Similarity](https://mlsanity.com/paper/2607.24880/) (2026)
- [Revisiting Predictive Process Monitoring in the Age of Foundation Models: A Comparative Study of Sequence, Tabular, and LLM Approaches](https://mlsanity.com/paper/2607.27797/) (2026)
- [Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths](https://mlsanity.com/paper/2608.18919/) (2026)
- [Benchmarking Attention for Tabular Foundation Models](https://mlsanity.com/paper/2609.31306/) (2026)

## Abstract (arXiv)

Tabular in-context learning (ICL) has recently achieved state-of-the-art (SOTA) performance on several tabular prediction tasks. Previously restricted to classification problems on small tables, recent advances such as TabPFN and TabICL have extended its use to larger datasets. Although current table-native ICL architectures are architecturally efficient and well-adapted to tabular data structures, their exclusive training on synthetic data limits their ability to fully leverage the rich semantics and world knowledge contained in real-world tabular data. At the other end of the spectrum, tabular ICL models based on pretrained large language models such as TabuLa-8B integrate deep semantic understanding and world knowledge but are only able to make use of a small amount of context due to inherent architectural limitations. With the aim to combine the best of both these worlds, we introduce ConTextTab, integrating semantic understanding and alignment into a table-native ICL framework. By employing specialized embeddings for different data modalities and by training on large-scale real-world tabular data, our model is competitive with SOTA across a broad set of benchmarks while setting a new standard on the semantically rich CARTE benchmark. Code and model checkpoints are available at: https://github.com/SAP-samples/sap-rpt-1-oss.