mlsanityData to 7 Oct 2026

A Survey on Self-Supervised Learning for Non-Sequential Tabular Data

arXiv 2402.01204 ↗First posted 2 Feb 2024cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has been incorporated into many state-of-the-art models in various domains, where SSL defines pretext tasks based on unlabeled datasets to learn contextualized and robust representations. Recently, SSL has become a new trend in exploring the representation learning capability in the realm of tabular data, which is more challenging due to not having explicit relations for learning descriptive representations. This survey aims to systematically review and summarize the recent progress and challenges of SSL for non-sequential tabular data (SSL4NS-TD). We first present a formal definition of NS-TD and clarify its correlation to related studies. Then, these approaches are categorized into three groups - predictive learning, contrastive learning, and hybrid learning, with their motivations and strengths of representative methods in each direction. Moreover, application issues of SSL4NS-TD are presented, including automatic data engineering, cross-table transferability, and domain knowledge integration. In addition, we elaborate on existing benchmarks and datasets for NS-TD applications to analyze the performance of existing tabular models. Finally, we discuss the challenges of SSL4NS-TD and provide potential directions for future research. We expect our work to be useful in terms of encouraging more research on lowering the barrier to entry SSL for the tabular domain, and of improving the foundations for implicit tabular data.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

43 of 58 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  6. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  7. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  8. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  9. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  10. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  12. 2020A Survey on Contrastive Self-supervised Learningcited by 553
31 more
  1. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
  2. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 398
  3. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 368
  4. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  5. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  6. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  7. 2023A Cookbook of Self-Supervised Learningcited by 248
  8. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 218
  9. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 200
  10. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  11. 2017OpenML Benchmarking Suitescited by 169
  12. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  13. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  14. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  15. 2021Unsupervised Speech Recognitioncited by 147
  16. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  17. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  18. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  19. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 101
  20. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  21. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  22. 2022Transfer Learning with Deep Tabular Modelscited by 52
  23. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 48
  24. 2023Multi-Modal Self-Supervised Learning for Recommendationcited by 44
  25. 2024SwitchTab: Switched Autoencoders Are Effective Tabular Learnerscited by 37
  26. 2023UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Sciencecited by 28
  27. 2023A Survey on User Behavior Modeling in Recommender Systemscited by 26
  28. 2023ReConTab: Regularized Contrastive Representation Learning for Tabular Datacited by 24
  29. 2023Self-supervised Representation Learning From Random Data Projectorscited by 14
  30. 2023DoRA: Domain-Based Self-Supervised Learning Framework for Low-Resource Real Estate Appraisalcited by 5
  31. 2023Enhancing Unsupervised Speech Recognition with Diffusion GANscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 4; 2026: 4.

15 papers.

Sort:

By area:

  1. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  2. 2024A Survey on Deep Tabular Learningcited by 18
  3. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 15
  4. 2024A Survey of the Self Supervised Learning Mechanisms for Vision Transformerscited by 8
  5. 2024GoNoGo: An Efficient LLM-based Multi-Agent System for Streamlining Automotive Software Release Decision-Makingcited by 6
  6. 2024TabMDA: Tabular Manifold Data Augmentation for Any Classifier using Transformers with In-context Subsettingcited by 5
  7. 2024Scalable Representation Learning for Multimodal Tabular Transactionscited by 5
  8. 2025Self-supervised learning on gene expression datacited by 1
  9. 2026LUCoS: Latent Unsupervised Context Selection for Tabular Foundation Modelscited by 1
  10. 2024APAR: Modeling Irregular Target Functions in Tabular Regression via Arithmetic-Aware Pre-Training and Adaptive-Regularized Fine-Tuningcited by 0
  11. 2024ORIGAMI: A generative transformer architecture for predictions from semi-structured datacited by 0
  12. 2025FairContrast: Enhancing Fairness through Contrastive learning and Customized Augmenting Methods on Tabular Datacited by 0
3 more
  1. 2026QuAIL: Quality-Aware Inertial Learning for Robust Training under Data Corruptioncited by 0
  2. 2026Beyond Statistical Co-occurrence: Unlocking Intrinsic Semantics for Tabular Data Clusteringcited by 0
  3. 2026ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Datacited by 0