mlsanityData to 14 Oct 2026

Revisiting Pretraining Objectives for Tabular Deep Learning

arXiv 2207.03208 ↗First posted 7 Jul 2022cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Recent deep learning models for tabular data currently compete with the traditional ML models based on decision trees (GBDT). Unlike GBDT, deep models can additionally benefit from pretraining, which is a workhorse of DL for vision and NLP. For tabular problems, several pretraining methods were proposed, but it is not entirely clear if pretraining provides consistent noticeable improvements and what method should be used, since the methods are often not compared to each other or comparison is limited to the simplest MLP architectures. In this work, we aim to identify the best practices to pretrain tabular DL models that can be universally applied to different datasets and architectures. Among our findings, we show that using the object target labels during the pretraining stage is beneficial for the downstream performance and advocate several target-aware pretraining objectives. Overall, our experiments demonstrate that properly performed pretraining significantly increases the performance of tabular DL models, which often leads to their superiority over GBDTs.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 7 Jul 2022.

Builds on

28 of 41 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  6. 2020Supervised Contrastive Learningcited by 2,829
  7. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  8. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  9. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  10. 2017Self-Normalizing Neural Networkscited by 598
  11. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  12. 2014OpenML: networked science in machine learningcited by 526
16 more
  1. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  2. 2017Deep & Cross Network for Ad Click Predictionscited by 393
  3. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 320
  4. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
  5. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 203
  6. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  7. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
  8. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  9. 2013Introducing LETOR 4.0 Datasetscited by 121
  10. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  11. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 113
  12. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 104
  13. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 101
  14. 2020The Tree Ensemble Layer: Differentiability meets Conditional Computationcited by 57
  15. 2020Gradient Boosting Neural Networks: GrowNetcited by 46
  16. 2021Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domaincited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 14; 2025: 9; 2026: 5.

43 papers.

Sort:

  1. 2023A Cookbook of Self-Supervised Learningcited by 250
  2. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  3. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
  4. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 113
  5. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  6. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  7. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 41
  8. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  9. 2023ExcelFormer: A neural network surpassing GBDTs on tabular datacited by 34
  10. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 33
  11. 2023TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specializationcited by 32
  12. 2024Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Latercited by 30
31 more
  1. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 25
  2. 2023A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learningcited by 24
  3. 2024A Survey on Deep Tabular Learningcited by 18
  4. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 18
  5. 2023FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Datacited by 14
  6. 2025On Finetuning Tabular Foundation Modelscited by 13
  7. 2024Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domainscited by 12
  8. 2024Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarkingcited by 9
  9. 2025Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformercited by 8
  10. 2023Reckoning with the Disagreement Problem: Explanation Consensus as a Training Objectivecited by 6
  11. 2023Homological Convolutional Neural Networkscited by 4
  12. 2023Scaling Experiments in Self-Supervised Cross-Table Representation Learningcited by 4
  13. 2025A Pre-training Framework for Relational Data with Information-theoretic Principlescited by 4
  14. 2023TabGSL: Graph Structure Learning for Tabular Data Predictioncited by 3
  15. 2024Tree-Regularized Tabular Embeddingscited by 3
  16. 2024Transformers with Stochastic Competition for Tabular Data Modellingcited by 3
  17. 2024mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Designcited by 3
  18. 2025TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimizationcited by 2
  19. 2025Unveiling the Role of Data Uncertainty in Tabular Deep Learningcited by 2
  20. 2023Tabular Machine Learning Methods for Predicting Gas Turbine Emissionscited by 1
  21. 2023Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcitycited by 1
  22. 2024Fine-grained Attention in Hierarchical Transformers for Tabular Time-seriescited by 1
  23. 2024A Neural Network Alternative to Tree-based Modelscited by 1
  24. 2025Improving Deep Tabular Learningcited by 1
  25. 2026Deep Tabular Representation Correctorcited by 1
  26. 2023LEURN: Learning Explainable Univariate Rules with Neural Networkscited by 0
  27. 2024Challenging Gradient Boosted Decision Trees with Tabular Transformers for Fraud Detection at Booking.comcited by 0
  28. 2026When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learningcited by 0
  29. 2026InFlux++: Real and Synthetic Data for Estimating Dynamic Camera Intrinsicscited by 0
  30. 2026The Importance of Encoder Choice:A Tabular-Image Studycited by 0
  31. 2026Solving In-Table Prediction Problems by Deep Neural Networks with Performance Evaluation Using Synthetic Datacited by 0