mlsanityData to 14 Oct 2026

Scaling Experiments in Self-Supervised Cross-Table Representation Learning

arXiv 2309.17339 ↗First posted 29 Sep 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

To analyze the scaling potential of deep tabular representation learning models, we introduce a novel Transformer-based architecture specifically tailored to tabular data and cross-table representation learning by utilizing table-specific tokenizers and a shared Transformer backbone. Our training approach encompasses both single-table and cross-table models, trained via missing value imputation through a self-supervised masked cell recovery objective. To understand the scaling behavior of our method, we train models of varying sizes, ranging from approximately 104 to 107 parameters. These models are trained on a carefully curated pretraining dataset, consisting of 135M training tokens sourced from 76 diverse datasets. We assess the scaling of our architecture in both single-table and cross-table pretraining setups by evaluating the pretrained models using linear probing on a curated set of benchmark datasets and comparing the results with conventional baselines.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 40 of its 52 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

40 of 52 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  6. 2023BloombergGPT: A Large Language Model for Financecited by 939
  7. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  8. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  9. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  10. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  11. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  12. 2014OpenML: networked science in machine learningcited by 526
28 more
  1. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  2. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
  3. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  4. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  5. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
  6. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  7. 2022Language Models are Realistic Tabular Data Generatorscited by 294
  8. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  9. 2018CatBoost: gradient boosting with categorical features supportcited by 208
  10. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  11. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  12. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
  13. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  14. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  15. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  16. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  17. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 104
  18. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 91
  19. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  20. 2023Tokenization and the Noiseless Channelcited by 59
  21. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  22. 2022Transfer Learning with Deep Tabular Modelscited by 52
  23. 2022Reasoning Like Program Executorscited by 45
  24. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  25. 2022StruBERT: Structure-aware BERT for Table Search and Matchingcited by 24
  26. 2021Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domaincited by 20
  27. 2022STable: Table Generation Framework for Encoder-Decoder Modelscited by 15
  28. 2023TabRet: Pre-training Transformer-based Tabular Models for Unseen Columnscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
  2. 2024Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolutioncited by 19
  3. 2026PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Modelscited by 19
  4. 2025Imputation-free Learning of Tabular Data with Missing Values using Incremental Feature Partitions in Transformercited by 1