mlsanityData to 7 Oct 2026

TabRet: Pre-training Transformer-based Tabular Models for Unseen Columns

arXiv 2303.15747 ↗First posted 28 Mar 2023cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

We present TabRet, a pre-trainable Transformer-based model for tabular data. TabRet is designed to work on a downstream task that contains columns not seen in pre-training. Unlike other methods, TabRet has an extra learning step before fine-tuning called retokenizing, which calibrates feature embeddings based on the masked autoencoding loss. In experiments, we pre-trained TabRet with a large collection of public health surveys and fine-tuned it on classification tasks in healthcare, and TabRet achieved the best AUC performance on four datasets. In addition, an ablation study shows retokenizing and random shuffle augmentation of columns during pre-training contributed to performance gains. The code is available at https://github.com/pfnet-research/tabret .

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

21 of 26 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2016Layer Normalizationcited by 5,579
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  12. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
9 more
  1. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  3. 2021A Survey of Transformerscited by 399
  4. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  5. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  6. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  7. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  8. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  9. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 3; 2026: 2.

14 papers.

Sort:

By area:

  1. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  2. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  3. 2024Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolutioncited by 19
  4. 2024CTSyn: A Foundation Model for Cross Tabular Data Generationcited by 13
  5. 2023Rethinking Data Augmentation for Tabular Data in Deep Learningcited by 10
  6. 2023Unlocking the Transferability of Tokens in Deep Models for Tabular Datacited by 8
  7. 2024Data Science with LLMs and Interpretable Modelscited by 8
  8. 2023LLMs Understand Glass-Box Models, Discover Surprises, and Suggest Repairscited by 7
  9. 2024Learning a Decision Tree Algorithm with Transformerscited by 6
  10. 2024Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMscited by 5
  11. 2023Scaling Experiments in Self-Supervised Cross-Table Representation Learningcited by 4
  12. 2025TabKAN: Advancing Tabular Data Analysis using Kolmogorov-Arnold Networkcited by 3
2 more
  1. 2026Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Datacited by 0
  2. 2026A Comparative Framework for Evaluating Foundation Models on Tabular Data: A Case Study in Healthcarecited by 0