mlsanityData to 14 Oct 2026

SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

arXiv 2106.01342 ↗First posted 2 Jun 2021cs.LG cs.AI stat.MLcited by 319

Not on the map. Why?

Abstract (arXiv)

Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even outperforms gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over a variety of benchmark tasks.

Key facts. Cited by 319 papers in the map (data through 14 Oct 2026). Builds on 28 of its 59 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

28 of 59 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2017Graph Attention Networkscited by 8,923
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2016Layer Normalizationcited by 5,669
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
16 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  6. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  7. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  8. 2014OpenML: networked science in machine learningcited by 526
  9. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  10. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  11. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
  12. 2019Axial Attention in Multidimensional Transformerscited by 358
  13. 2018CatBoost: gradient boosting with categorical features supportcited by 208
  14. 2021TABBIE: Pretrained Representations of Tabular Datacited by 111
  15. 2020Tabular Transformers for Modeling Multivariate Time Seriescited by 77
  16. 2019Learning Semantic Annotations for Tabular Datacited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 28; 2023: 61; 2024: 70; 2025: 73; 2026: 82.

319 papers.

Sort:

By area:

  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  2. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  3. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  4. 2022Language Models are Realistic Tabular Data Generatorscited by 294
  5. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  6. 2023A Cookbook of Self-Supervised Learningcited by 250
  7. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  8. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 199
  9. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  10. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  11. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  12. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
38 more
  1. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  2. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 113
  3. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 108
  4. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 106
  5. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  6. 2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 81
  7. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 77
  8. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 76
  9. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
  10. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 74
  11. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 72
  12. 2022Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matchingcited by 68
  13. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 66
  14. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  15. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 59
  16. 2024Making Pre-trained Language Models Great on Tabular Predictioncited by 57
  17. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 53
  18. 2022Transfer Learning with Deep Tabular Modelscited by 52
  19. 2024TALENT: A Tabular Analytics and Learning Toolboxcited by 52
  20. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 49
  21. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 49
  22. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  23. 2025Griffin: Towards a Graph-Centric Relational Database Foundation Modelcited by 47
  24. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  25. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  26. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 41
  27. 2024A Comprehensive Survey on Data Augmentationcited by 40
  28. 2024SwitchTab: Switched Autoencoders Are Effective Tabular Learnerscited by 38
  29. 2023Trompt: Towards a Better Deep Neural Network for Tabular Datacited by 36
  30. 2023HYTREL: Hypergraph-enhanced Tabular Data Representation Learningcited by 36
  31. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  32. 2023ExcelFormer: A neural network surpassing GBDTs on tabular datacited by 34
  33. 2024Mixture of In-Context Prompters for Tabular PFNscited by 33
  34. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 33
  35. 2024PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learningcited by 32
  36. 2024TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Datacited by 32
  37. 2022PTab: Using the Pre-trained Language Model for Modeling Tabular Datacited by 30
  38. 2024Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Latercited by 30

50 most cited of 319 All 319 (JSON)