mlsanityData to 14 Oct 2026

Transfer Learning with Deep Tabular Models

arXiv 2206.15306 ↗First posted 30 Jun 2022cs.LG stat.MLcited by 52

Not on the map. Why?

Abstract (arXiv)

Recent work on deep learning for tabular data demonstrates the strong performance of deep tabular models, often bridging the gap between gradient boosted decision trees and neural networks. Accuracy aside, a major advantage of neural models is that they learn reusable features and are easily fine-tuned in new domains. This property is often exploited in computer vision and natural language applications, where transfer learning is indispensable when task-specific training data is scarce. In this work, we demonstrate that upstream data gives tabular neural networks a decisive advantage over widely used GBDT models. We propose a realistic medical diagnosis benchmark for tabular transfer learning, and we present a how-to guide for using upstream data to boost performance with a variety of tabular neural network architectures. Finally, we propose a pseudo-feature method for cases where the upstream and downstream feature sets differ, a tabular-specific problem widespread in real-world applications. Our code is available at https://github.com/LevinRoman/tabular-transfer-learning .

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 51 of its 103 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

51 of 103 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  10. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  11. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  12. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2018YOLOv3: An Incremental Improvementcited by 2,659
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  7. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  9. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  10. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  11. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  12. 2017Self-Normalizing Neural Networkscited by 598
  13. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  14. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  15. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  16. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  17. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  18. 2017Deep & Cross Network for Ad Click Predictionscited by 393
  19. 2018GAIN: Missing Data Imputation using Generative Adversarial Netscited by 352
  20. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 327
  21. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 320
  22. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
  23. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
  24. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 203
  25. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  26. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
  27. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  28. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 113
  29. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 104
  30. 2018Deep Neural Decision Treescited by 72
  31. 2020The Tree Ensemble Layer: Differentiability meets Conditional Computationcited by 57
  32. 2020Gradient Boosting Neural Networks: GrowNetcited by 46
  33. 2021Locally Sparse Neural Networks for Tabular Biomedical Datacited by 34
  34. 2021Neighborhood Contrastive Learning Applied to Online Patient Monitoringcited by 32
  35. 2022Hopular: Modern Hopfield Networks for Tabular Datacited by 18
  36. 2020Joint Liver Lesion Segmentation and Classification via Transfer Learningcited by 14
  37. 2021Simple Modifications to Improve Tabular Neural Networkscited by 14
  38. 2020Adapted tree boosting for Transfer Learningcited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 23; 2024: 13; 2025: 9; 2026: 5.

52 papers.

Sort:

By area:

  1. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  2. 2023A Cookbook of Self-Supervised Learningcited by 250
  3. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  4. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  5. 2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 81
  6. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 76
  7. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  8. 2023Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimescited by 52
  9. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 49
  10. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  11. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  12. 2023TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specializationcited by 32
38 more
  1. 2023MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinementcited by 24
  2. 2023From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Modelscited by 24
  3. 2023A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learningcited by 24
  4. 2025Table Foundation Models: on knowledge pre-training for tabular learningcited by 19
  5. 2023ProtoGate: Prototype-based Neural Networks with Global-to-local Feature Selection for Tabular Biomedical Datacited by 18
  6. 2023Simplifying Neural Network Training Under Class Imbalancecited by 18
  7. 2024A Survey on Deep Tabular Learningcited by 18
  8. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 18
  9. 2024Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learningcited by 17
  10. 2023Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Seriescited by 16
  11. 2024TabEBM: A Tabular Data Augmentation Method with Distinct Class-Specific Energy-Based Modelscited by 16
  12. 2024A Survey on Self-Supervised Learning for Non-Sequential Tabular Datacited by 15
  13. 2023FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Datacited by 14
  14. 2023Unlocking the Transferability of Tokens in Deep Models for Tabular Datacited by 8
  15. 2025Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformercited by 8
  16. 2022GCondNet: A Novel Method for Improving Neural Networks on Small High-Dimensional Tabular Datacited by 6
  17. 2023Reckoning with the Disagreement Problem: Explanation Consensus as a Training Objectivecited by 6
  18. 2024Improving LLM Group Fairness on Tabular Data via In-Context Learningcited by 6
  19. 2023Deep Active Learning in the Presence of Label Noise: A Surveycited by 5
  20. 2023Adapting Pretrained Language Models for Solving Tabular Prediction Problems in the Electronic Health Recordcited by 5
  21. 2023Scaling Experiments in Self-Supervised Cross-Table Representation Learningcited by 4
  22. 2024LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systemscited by 4
  23. 2025Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizescited by 4
  24. 2025The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Taskscited by 4
  25. 2023Revisiting Self-Training with Regularized Pseudo-Labeling for Tabular Datacited by 2
  26. 2024pTNAS: Progressive Neural Architecture Search for Tabular Datacited by 2
  27. 2024Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Datacited by 2
  28. 2025LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domainscited by 2
  29. 2023Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcitycited by 1
  30. 2025LEKA:LLM-Enhanced Knowledge Augmentationcited by 1
  31. 2026Active Tabular Augmentation via Policy-Guided Diffusion Inpaintingcited by 1
  32. 2023LEURN: Learning Explainable Univariate Rules with Neural Networkscited by 0
  33. 2023Global Layers: Non-IID Tabular Federated Learningcited by 0
  34. 2024Boosting gets full Attention for Relational Learningcited by 0
  35. 2024Challenging Gradient Boosted Decision Trees with Tabular Transformers for Fraud Detection at Booking.comcited by 0
  36. 2025Probabilistic Pretraining for Neural Regressioncited by 0
  37. 2026LakeHopper: Knowledge-Aware Adaptation of Column Type Annotators across Data Lakescited by 0
  38. 2026Closing the gap on tabular data with Fourier and Implicit Categorical Featurescited by 0

50 most cited of 52 All 52 (JSON)