mlsanityData to 7 Oct 2026

PORTAL: Scalable Tabular Foundation Models via Content-Specific Tokenization

arXiv 2410.13516 ↗First posted 17 Oct 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning on tabular data seeks to apply advances from natural language and image domains to the diverse domain of tables. However, current techniques often struggle with integrating multi-domain data and require data cleaning or specific structural requirements, limiting the scalability of pre-training datasets. We introduce PORTAL (Pretraining One-Row-at-a-Time for All tabLes), a framework that handles various data modalities without the need for cleaning or preprocessing. This simple yet powerful approach can be effectively pre-trained on online-collected datasets and fine-tuned to match state-of-the-art methods on complex classification and regression tasks. This work offers a practical advancement in self-supervised learning for large-scale tabular data.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 14 of its 17 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

14 of 17 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  6. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  7. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  8. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 368
  9. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  10. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  11. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 69
  12. 2024Making Pre-trained Language Models Great on Tabular Predictioncited by 56
2 more
  1. 2023UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Sciencecited by 28
  2. 2023Vectorizing string entries for data processing on tables: when are larger language models better?cited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 4.

13 papers.

Sort:

  1. 2025Relational Transformer: Toward Zero-Shot Foundation Models for Relational Datacited by 32
  2. 2025ConTextTab: A Semantics-Aware Tabular In-Context Learnercited by 31
  3. 2026RelBench v2: A Large-Scale Benchmark and Repository for Relational Datacited by 17
  4. 2025Table Foundation Models: on knowledge pre-training for tabular learningcited by 15
  5. 2025Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformercited by 8
  6. 2025Position: Foundation Models for Tabular Data within Systemic Contexts Need Groundingcited by 5
  7. 2025Synthetic Tabular Data Detection In the Wildcited by 2
  8. 2025RELATE: A Schema-Agnostic Perceiver Encoder for Multimodal Relational Graphscited by 1
  9. 2026Language Model Representations for Efficient Few-Shot Tabular Classificationcited by 1
  10. 2025Datum-wise Transformer for Synthetic Tabular Data Detection in the Wildcited by 0
  11. 2025Robust Detection of Synthetic Tabular Data under Schema Variabilitycited by 0
  12. 2026SALT-KG: A Benchmark for Semantics-Aware Learning on Enterprise Tablescited by 0
1 more
  1. 2026Embedding World Knowledge into Tabular Models: Towards Best Practices for Embedding Pipeline Designcited by 0