# PORTAL: Scalable Tabular Foundation Models via Content-Specific Tokenization

arXiv: https://arxiv.org/abs/2410.13516 · HTML: https://mlsanity.com/paper/2410.13516/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [CARTE: Pretraining and Transfer for Tabular Learning](https://mlsanity.com/paper/2402.16785/) (2024)
- [Making Pre-trained Language Models Great on Tabular Prediction](https://mlsanity.com/paper/2403.01841/) (2024)
- [UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science](https://mlsanity.com/paper/2307.09249/) (2023)
- [Vectorizing string entries for data processing on tables: when are larger language models better?](https://mlsanity.com/paper/2312.09634/) (2023)

## Built on by (most cited first, 14 of 14)

- [Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data](https://mlsanity.com/paper/2510.06377/) (2025)
- [ConTextTab: A Semantics-Aware Tabular In-Context Learner](https://mlsanity.com/paper/2506.10707/) (2025)
- [RelBench v2: A Large-Scale Benchmark and Repository for Relational Data](https://mlsanity.com/paper/2602.12606/) (2026)
- [Table Foundation Models: on knowledge pre-training for tabular learning](https://mlsanity.com/paper/2505.14415/) (2025)
- [Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer](https://mlsanity.com/paper/2502.04573/) (2025)
- [Position: Foundation Models for Tabular Data within Systemic Contexts Need Grounding](https://mlsanity.com/paper/2505.19825/) (2025)
- [Synthetic Tabular Data Detection In the Wild](https://mlsanity.com/paper/2503.01937/) (2025)
- [Language Model Representations for Efficient Few-Shot Tabular Classification](https://mlsanity.com/paper/2602.15844/) (2026)
- [Embedding World Knowledge into Tabular Models: Towards Best Practices for Embedding Pipeline Design](https://mlsanity.com/paper/2603.17737/) (2026)
- [RELATE: A Schema-Agnostic Perceiver Encoder for Multimodal Relational Graphs](https://mlsanity.com/paper/2510.19954/) (2025)
- [Datum-wise Transformer for Synthetic Tabular Data Detection in the Wild](https://mlsanity.com/paper/2504.08829/) (2025)
- [Robust Detection of Synthetic Tabular Data under Schema Variability](https://mlsanity.com/paper/2509.00092/) (2025)
- [SALT-KG: A Benchmark for Semantics-Aware Learning on Enterprise Tables](https://mlsanity.com/paper/2601.07638/) (2026)
- [RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions](https://mlsanity.com/paper/2610.17867/) (2026)

## Abstract (arXiv)

Self-supervised learning on tabular data seeks to apply advances from natural language and image domains to the diverse domain of tables. However, current techniques often struggle with integrating multi-domain data and require data cleaning or specific structural requirements, limiting the scalability of pre-training datasets. We introduce PORTAL (Pretraining One-Row-at-a-Time for All tabLes), a framework that handles various data modalities without the need for cleaning or preprocessing. This simple yet powerful approach can be effectively pre-trained on online-collected datasets and fine-tuned to match state-of-the-art methods on complex classification and regression tasks. This work offers a practical advancement in self-supervised learning for large-scale tabular data.