mlsanityData to 7 Oct 2026

Why Tabular Foundation Models Should Be a Research Priority

arXiv 2405.01147 ↗First posted 2 May 2024cs.LGcited by 103

Not on the map. Why?

Abstract (arXiv)

Recent text and image foundation models are incredibly impressive, and these models are attracting an ever-increasing portion of research resources. In this position piece we aim to shift the ML research community's priorities ever so slightly to a different modality: tabular data. Tabular data is the dominant modality in many fields, yet it is given hardly any research attention and significantly lags behind in terms of scale and power. We believe the time is now to start developing tabular foundation models, or what we coin a Large Tabular Model (LTM). LTMs could revolutionise the way science and ML use tabular data: not as single datasets that are analyzed in a vacuum, but contextualized with respect to related datasets. The potential impact is far-reaching: from few-shot tabular models to automating data science; from out-of-distribution synthetic data to empowering multidisciplinary scientific discovery. We intend to excite reflections on the modalities we study, and convince some researchers to study large tabular models.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Builds on 63 of its 102 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

63 of 102 references.

References · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  12. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
38 more
  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  2. 2023Textbooks Are All You Needcited by 561
  3. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
  4. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  5. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  6. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  7. 2018Assessing Generative Models via Precision and Recallcited by 401
  8. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 398
  9. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  10. 2020Calibration of Pre-trained Transformerscited by 307
  11. 2015A note on the evaluation of generative modelscited by 291
  12. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  13. 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 278
  14. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  15. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  16. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 218
  17. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  18. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 193
  19. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  20. 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 165
  21. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 161
  22. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  23. 2019Learning from Synthetic Data for Crowd Counting in the Wildcited by 148
  24. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  25. 2023Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineeringcited by 125
  26. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 122
  27. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 121
  28. 2023Compositional Foundation Models for Hierarchical Planningcited by 92
  29. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  30. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  31. 2020Tabular Transformers for Modeling Multivariate Time Seriescited by 77
  32. 2022Synthetic Data -- what, why and how?cited by 66
  33. 2023Membership Inference Attacks against Synthetic Data through Overfitting Detectioncited by 66
  34. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 64
  35. 2018Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numberscited by 42
  36. 2023TabuLa: Harnessing Language Models for Tabular Data Synthesiscited by 42
  37. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 31
  38. 2023Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Datacited by 29

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 49; 2026: 39.

103 papers.

Sort:

By area:

  1. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  2. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  3. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 73
  4. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 67
  5. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 58
  6. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  7. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 33
  8. 2025ConTextTab: A Semantics-Aware Tabular In-Context Learnercited by 31
  9. 2025Foundation Models for Causal Inference via Prior-Data Fitted Networkscited by 30
  10. 2025Turning Tabular Foundation Models into Graph Foundation Modelscited by 30
  11. 2024Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Datacited by 25
  12. 2024TabEBM: A Tabular Data Augmentation Method with Distinct Class-Specific Energy-Based Modelscited by 15
38 more
  1. 2025Table Foundation Models: on knowledge pre-training for tabular learningcited by 15
  2. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 15
  3. 2025Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learningcited by 15
  4. 2024CTSyn: A Foundation Model for Cross Tabular Data Generationcited by 13
  5. 2025TabPFN-Wide: Continued Pre-Training for Extreme Feature Countscited by 13
  6. 2025Bringing Graphs to the Table: Zero-shot Node Classification via Tabular Foundation Modelscited by 12
  7. 2025TabPFN: One Model to Rule Them All?cited by 10
  8. 2025Early Stopping Tabular In-Context Learningcited by 10
  9. 2025Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challengescited by 10
  10. 2025TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Modelscited by 10
  11. 2024Large Language Models Engineer Too Many Simple Features For Tabular Datacited by 8
  12. 2025LLM-TabLogic: Preserving Inter-Column Logical Relationships in Synthetic Tabular Data via Prompt-Guided Latent Diffusioncited by 8
  13. 2026From Zero to Hero: Advancing Zero-Shot Foundation Models for Tabular Outlier Detectioncited by 8
  14. 2024IM-Context: In-Context Learning for Imbalanced Regression Taskscited by 7
  15. 2024Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Frameworkcited by 7
  16. 2024LaTable: Towards Large Tabular Modelscited by 7
  17. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 7
  18. 2025Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Modelscited by 7
  19. 2025Serialized EHR make for good text representationscited by 7
  20. 2025Table as a Modality for Large Language Modelscited by 7
  21. 2024FairJob: A Real-World Dataset for Fairness in Online Systemscited by 6
  22. 2024GoNoGo: An Efficient LLM-based Multi-Agent System for Streamlining Automotive Software Release Decision-Makingcited by 6
  23. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 6
  24. 2025Orion-Bix: Bi-Axial Attention for Tabular In-Context Learningcited by 6
  25. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 6
  26. 2024Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMscited by 5
  27. 2024Scalable Representation Learning for Multimodal Tabular Transactionscited by 5
  28. 2025Position: Foundation Models for Tabular Data within Systemic Contexts Need Groundingcited by 5
  29. 2025Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settingscited by 5
  30. 2026End-to-End Compression for Tabular Foundation Modelscited by 5
  31. 2025Tabby: A Language Model Architecture for Tabular and Structured Data Synthesiscited by 4
  32. 2025GateLens: A Reasoning-Enhanced LLM Agent for Automotive Software Release Analyticscited by 4
  33. 2025Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learningcited by 4
  34. 2026Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretrainingcited by 4
  35. 2026Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanismscited by 4
  36. 2026TabDPT-Turbo: Efficient In-Context Learning for Tabular Predictioncited by 4
  37. 2025Creating Artificial Students that Never Existed: Leveraging Large Language Models and CTGANs for Synthetic Data Generationcited by 3
  38. 2025Synthetic Data Generation for Augmenting Small Samplescited by 3

50 most cited of 103 All 103 (JSON)