mlsanityData to 13 Oct 2026

XTab: Cross-table Pretraining for Tabular Transformers

arXiv 2305.06090 ↗First posted 10 May 2023cs.LGcited by 81

Not on the map. Why?

Abstract (arXiv)

The success of self-supervised learning in computer vision and natural language processing has motivated pretraining methods on tabular data. However, most existing tabular self-supervised learning models fail to leverage information across multiple data tables and cannot generalize to new tables. In this work, we introduce XTab, a framework for cross-table pretraining of tabular transformers on datasets from various domains. We address the challenge of inconsistent column types and quantities among tables by utilizing independent featurizers and using federated learning to pretrain the shared component. Tested on 84 tabular prediction tasks from the OpenML-AutoML Benchmark (AMLB), we show that (1) XTab consistently boosts the generalizability, learning speed, and performance of multiple tabular transformers, (2) by pretraining FT-Transformer via XTab, we achieve superior performance than other state-of-the-art tabular deep learning models on various tasks such as regression, binary, and multiclass classification.

Key facts. Cited by 81 papers in the map (data through 13 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

32 of 39 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  7. 2016Layer Normalizationcited by 5,624
  8. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,143
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,601
  11. 2020GLU Variants Improve Transformercited by 1,759
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,384
20 more
  1. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,227
  2. 2021Revisiting Deep Learning Models for Tabular Datacited by 733
  3. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 652
  4. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 397
  5. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 375
  6. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 315
  7. 2018CatBoost: gradient boosting with categorical features supportcited by 208
  8. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  9. 2020fastai: A Layered API for Deep Learningcited by 183
  10. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 153
  11. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  12. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 102
  13. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 65
  14. 2021Fastformer: Additive Attention Can Be All You Needcited by 61
  15. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
  16. 2022AMLB: an AutoML Benchmarkcited by 55
  17. 2022Transfer Learning with Deep Tabular Modelscited by 52
  18. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 47
  19. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  20. 2020Practical and sample efficient zero-shot HPOcited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 10; 2024: 25; 2025: 31; 2026: 15.

81 papers.

Sort:

By area:

  1. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 272
  2. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  3. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 101
  4. 2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 79
  5. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 71
  6. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 69
  7. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 69
  8. 2024Making Pre-trained Language Models Great on Tabular Predictioncited by 56
  9. 2023Relational Deep Learning: Graph Representation Learning on Relational Databasescited by 55
  10. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoningcited by 48
  11. 2025Griffin: Towards a Graph-Centric Relational Database Foundation Modelcited by 43
  12. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 39
38 more
  1. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  2. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 35
  3. 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 32
  4. 2024PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learningcited by 32
  5. 2025ConTextTab: A Semantics-Aware Tabular In-Context Learnercited by 32
  6. 2024PTaRL: Prototype-based Tabular Representation Learning via Space Calibrationcited by 27
  7. 2023MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinementcited by 24
  8. 2023A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learningcited by 24
  9. 2025Relational Deep Learning: Challenges, Foundations and Next-Generation Architecturescited by 22
  10. 2024A Survey on Deep Tabular Learningcited by 18
  11. 2025Table Foundation Models: on knowledge pre-training for tabular learningcited by 16
  12. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 16
  13. 2024A Survey on Self-Supervised Learning for Non-Sequential Tabular Datacited by 15
  14. 2024T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Datacited by 14
  15. 2024CTSyn: A Foundation Model for Cross Tabular Data Generationcited by 13
  16. 2024PORTAL: Scalable Tabular Foundation Models via Content-Specific Tokenizationcited by 13
  17. 2025TabFlex: Scaling Tabular Learning to Millions with Linear Attentioncited by 13
  18. 2024Unlock the Potential of Large Language Models for Predictive Tabular Tasks in Data Science with Table-Specific Pretrainingcited by 12
  19. 2024Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domainscited by 12
  20. 2026Relational In-Context Learning via Synthetic Pre-training with Structural Priorcited by 12
  21. 2024Tabular Transfer Learning via Prompting LLMscited by 10
  22. 2023UniPredict: Large Language Models are Universal Tabular Classifierscited by 9
  23. 2023Unlocking the Transferability of Tokens in Deep Models for Tabular Datacited by 8
  24. 2025Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformercited by 8
  25. 2024Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Frameworkcited by 7
  26. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 7
  27. 2025LLM Meeting Decision Trees on Tabular Datacited by 7
  28. 2023Fine-Tuning the Retrieval Mechanism for Tabular Deep Learningcited by 6
  29. 2024Learning a Decision Tree Algorithm with Transformerscited by 6
  30. 2024Improving LLM Group Fairness on Tabular Data via In-Context Learningcited by 6
  31. 2025ZEUS: Zero-shot Embeddings for Unsupervised Separation of Tabular Datacited by 6
  32. 2024Scalable Representation Learning for Multimodal Tabular Transactionscited by 5
  33. 2025Position: Foundation Models for Tabular Data within Systemic Contexts Need Groundingcited by 5
  34. 2023Scaling Experiments in Self-Supervised Cross-Table Representation Learningcited by 4
  35. 2025Task-Agnostic Contrastive Pretraining for Relational Deep Learningcited by 4
  36. 2025EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognitioncited by 4
  37. 2025TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrievalcited by 4
  38. 2026TabSieve: Explicit In-Table Evidence Selection for Tabular Predictioncited by 4

50 most cited of 81 All 81 (JSON)