mlsanityData to 7 Oct 2026

Generative Table Pre-training Empowers Models for Tabular Prediction

arXiv 2305.09696 ↗First posted 16 May 2023cs.LG cs.AI cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

Recently, the topic of table pre-training has attracted considerable research interest. However, how to employ table pre-training to boost the performance of tabular prediction remains an open challenge. In this paper, we propose TapTap, the first attempt that leverages table pre-training to empower models for tabular prediction. After pre-training on a large corpus of real-world tabular data, TapTap can generate high-quality synthetic tables to support various applications on tabular data, including privacy protection, low resource regime, missing value imputation, and imbalanced classification. Extensive experiments on 12 datasets demonstrate that TapTap outperforms a total of 16 baselines in different scenarios. Meanwhile, it can be easily combined with various backbone models, including LightGBM, Multilayer Perceptron (MLP) and Transformer. Moreover, with the aid of table pre-training, models trained using synthetic data generated by TapTap can even compete with models using the original dataset on half of the experimental datasets, marking a milestone in the development of synthetic tabular data generation. The codes are available at https://github.com/ZhangTP1996/TapTap.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 45 of its 79 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

45 of 79 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  9. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  10. 2019Modeling Tabular data using Conditional GANcited by 695
  11. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  12. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
33 more
  1. 2014OpenML: networked science in machine learningcited by 523
  2. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  3. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  4. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  5. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  6. 2018GAIN: Missing Data Imputation using Generative Adversarial Netscited by 349
  7. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 326
  8. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 319
  9. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  10. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  11. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  12. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  13. 2017Generating Multi-label Discrete Patient Records using Generative Adversarial Networkscited by 197
  14. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  15. 2018Data Synthesis based on Generative Adversarial Networkscited by 184
  16. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  17. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  18. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  19. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  20. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  21. 2020TUTA: Tree-based Transformers for Generally Structured Table Pre-trainingcited by 100
  22. 2020Missing Data Imputation using Optimal Transportcited by 99
  23. 2022HyperImpute: Generalized Iterative Imputation with Automatic Model Selectioncited by 81
  24. 2020Understanding tables with intermediate pre-trainingcited by 68
  25. 2020VAEM: a Deep Generative Model for Heterogeneous Mixed Type Datacited by 66
  26. 2022Transfer Learning with Deep Tabular Modelscited by 52
  27. 2022Reasoning Like Program Executorscited by 45
  28. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  29. 2020RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparationcited by 27
  30. 2022Table-To-Text generation and pre-training with TabT5cited by 20
  31. 2018Airline Passenger Name Record Generation using Generative Adversarial Networkscited by 19
  32. 2020Google Dataset Search by the Numberscited by 17
  33. 2021Synthesising Multi-Modal Minority Samples for Tabular Datacited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 19; 2025: 23; 2026: 5.

55 papers.

Sort:

By area:

  1. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  2. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  3. 2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 79
  4. 2023Questioning the Survey Responses of Large Language Modelscited by 70
  5. 2024Making Pre-trained Language Models Great on Tabular Predictioncited by 56
  6. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  7. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  8. 2025A Comprehensive Survey of Synthetic Tabular Data Generationcited by 27
  9. 2023From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Modelscited by 22
  10. 2023Towards Cross-Table Masked Pretraining for Web Data Miningcited by 21
  11. 2024Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolutioncited by 19
  12. 2025TARGET: Benchmarking Table Retrieval for Generative Taskscited by 17
38 more
  1. 2024A Survey on Self-Supervised Learning for Non-Sequential Tabular Datacited by 15
  2. 2023The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against "Truly Anonymous" Synthetic Datasetscited by 14
  3. 2024CTSyn: A Foundation Model for Cross Tabular Data Generationcited by 13
  4. 2023Observatory: Characterizing Embeddings of Relational Tablescited by 12
  5. 2024Unlock the Potential of Large Language Models for Predictive Tabular Tasks in Data Science with Table-Specific Pretrainingcited by 12
  6. 2025Position: All Current Generative Fidelity and Diversity Metrics are Flawedcited by 12
  7. 2025LLM Embeddings for Deep Learning on Tabular Datacited by 11
  8. 2025The DCR Delusion: Measuring the Privacy Risk of Synthetic Datacited by 11
  9. 2025TabGen-ICL: Residual-Aware In-Context Example Selection for Tabular Data Generationcited by 10
  10. 2024Why LLMs Are Bad at Synthetic Table Generation (and what to do about it)cited by 9
  11. 2024Generating Realistic Tabular Data with Large Language Modelscited by 9
  12. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  13. 2024MALLM-GAN: Multi-Agent Large Language Model as Generative Adversarial Network for Synthesizing Tabular Datacited by 7
  14. 2025Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Modelscited by 7
  15. 2024GoNoGo: An Efficient LLM-based Multi-Agent System for Streamlining Automotive Software Release Decision-Makingcited by 6
  16. 2024Scalable Representation Learning for Multimodal Tabular Transactionscited by 5
  17. 2023Scaling Experiments in Self-Supervised Cross-Table Representation Learningcited by 4
  18. 2023Privacy Measurement in Tabular Synthetic Data: State of the Art and Future Research Directionscited by 4
  19. 2024P-TA: Using Proximal Policy Optimization to Enhance Tabular Data Augmentation via Large Language Modelscited by 4
  20. 2025Tabby: A Language Model Architecture for Tabular and Structured Data Synthesiscited by 4
  21. 2025Chain-of-Query: Unleashing the Power of LLMs in SQL-Aided Table Understanding via Multi-Agent Collaborationcited by 4
  22. 2025Large Language Models for Imbalanced Classification: Diversity makes the differencecited by 4
  23. 2024iTBLS: A Dataset of Interactive Conversations Over Tabular Informationcited by 3
  24. 2024Generating Tables from the Parametric Knowledge of Language Modelscited by 3
  25. 2024Team up GBDTs and DNNs: Advancing Efficient and Effective Tabular Prediction with Tree-hybrid MLPscited by 3
  26. 2025Zero-Shot Decision Tree Construction via Large Language Modelscited by 3
  27. 2025TabNSA: Native Sparse Attention for Efficient Tabular Data Learningcited by 2
  28. 2025Tabular Embeddings for Tables with Bi-Dimensional Hierarchical Metadata and Nestingcited by 1
  29. 2025Large Language Models as Attribution Regularizers for Efficient Model Trainingcited by 1
  30. 2025A Generalized Theory of Mixup for Structure-Preserving Synthetic Datacited by 1
  31. 2025Enabling Few-Shot Alzheimer's Disease Diagnosis on Biomarker Data with Tabular LLMscited by 1
  32. 2025Towards Universal Debiasing for Language Models-based Tabular Data Generationcited by 1
  33. 2025LLM as an Algorithmist: Enhancing Anomaly Detectors via Programmatic Synthesiscited by 1
  34. 2025Causal-Aware Tabular GANs with Reinforcement Learningcited by 1
  35. 2023An Adaptive Framework of Geographical Group-Specific Network on O2O Recommendationcited by 0
  36. 2024A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Ratiocited by 0
  37. 2025The Prompt is Mightier than the Examplecited by 0
  38. 2025StructSynth: Dependency Graphs as Generation Plans for Low-Data Tabular Synthesis with Language Modelscited by 0

50 most cited of 55 All 55 (JSON)