mlsanityData to 7 Oct 2026

Making Pre-trained Language Models Great on Tabular Prediction

arXiv 2403.01841 ↗First posted 4 Mar 2024cs.CL cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. However, due to the heterogeneity among tables, such DNN bonus is still far from being well exploited on tabular data prediction (e.g., regression or classification tasks). Condensing knowledge from diverse domains, language models (LMs) possess the capability to comprehend feature names from various tables, potentially serving as versatile learners in transferring knowledge across distinct tables and diverse prediction tasks, but their discrete text representation space is inherently incompatible with numerical feature values in tables. In this paper, we present TP-BERTa, a specifically pre-trained LM for tabular data prediction. Concretely, a novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names. Comprehensive experiments demonstrate that our pre-trained TP-BERTa leads the performance among tabular DNNs and is competitive with Gradient Boosted Decision Tree models in typical tabular data regime.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

29 of 42 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  6. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  12. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
17 more
  1. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  3. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 326
  4. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 319
  5. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  6. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  7. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 200
  8. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  9. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  10. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  11. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  12. 2023TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPTcited by 57
  13. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  14. 2021DANets: Deep Abstract Networks for Tabular Data Classification and Regressioncited by 50
  15. 2022T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interactioncited by 41
  16. 2023ExcelFormer: A neural network surpassing GBDTs on tabular datacited by 33
  17. 2023Towards Cross-Table Masked Pretraining for Web Data Miningcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 21; 2026: 17.

56 papers.

Sort:

By area:

  1. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 67
  2. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 51
  3. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoningcited by 48
  4. 2024TALENT: A Tabular Analytics and Learning Toolboxcited by 46
  5. 2025Griffin: Towards a Graph-Centric Relational Database Foundation Modelcited by 41
  6. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  7. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  8. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizerscited by 35
  9. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 33
  10. 2024TWIN-GPT: Digital Twins for Clinical Trials via Large Language Modelcited by 24
  11. 2024A Survey on Deep Tabular Learningcited by 18
  12. 2024HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protectioncited by 16
38 more
  1. 2025Table Foundation Models: on knowledge pre-training for tabular learningcited by 15
  2. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 15
  3. 2024CTSyn: A Foundation Model for Cross Tabular Data Generationcited by 13
  4. 2024PORTAL: Scalable Tabular Foundation Models via Content-Specific Tokenizationcited by 13
  5. 2025LLM Embeddings for Deep Learning on Tabular Datacited by 11
  6. 2024Tabular Transfer Learning via Prompting LLMscited by 10
  7. 2025Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformercited by 8
  8. 2025LLM Meeting Decision Trees on Tabular Datacited by 7
  9. 2025Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Modelscited by 7
  10. 2024Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMscited by 5
  11. 2024Scalable Representation Learning for Multimodal Tabular Transactionscited by 5
  12. 2026FORESTLLM: Large Language Models Make Random Forest Great on Few-shot Tabular Learningcited by 4
  13. 2026TabSieve: Explicit In-Table Evidence Selection for Tabular Predictioncited by 4
  14. 2024Team up GBDTs and DNNs: Advancing Efficient and Effective Tabular Prediction with Tree-hybrid MLPscited by 3
  15. 2024LLM Embeddings Improve Test-time Adaptation to Tabular Y|X-Shiftscited by 3
  16. 2024pTNAS: Progressive Neural Architecture Search for Tabular Datacited by 2
  17. 2025Architectural and Inferential Inductive Biases For Exchangeable Sequence Modelingcited by 2
  18. 2025Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboardcited by 2
  19. 2025TabDistill: Distilling Transformers into Neural Nets for Few-Shot Tabular Classificationcited by 2
  20. 2026MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Imagecited by 2
  21. 2024HySem: A context length optimized LLM pipeline for unstructured tabular extractioncited by 1
  22. 2024TREB: a BERT attempt for imputing tabular data imputationcited by 1
  23. 2025LLM as an Algorithmist: Enhancing Anomaly Detectors via Programmatic Synthesiscited by 1
  24. 2025Robust Tabular Foundation Modelscited by 1
  25. 2026Human-LLM Collaborative Feature Engineering for Tabular Datacited by 1
  26. 2026Task Expansion and Cross Refinement for Open-World Conditional Modelingcited by 1
  27. 2026BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classificationcited by 1
  28. 2024SysCaps: Language Interfaces for Simulation Surrogates of Complex Systemscited by 0
  29. 2024LBC: Language-Based-Classifier for Out-Of-Variable Generalizationcited by 0
  30. 2025Automatic Demonstration Selection for LLM-based Tabular Data Classificationcited by 0
  31. 2025Predictive Multimodal Modeling of Diagnoses and Treatments in EHRcited by 0
  32. 2025Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddingscited by 0
  33. 2025Can Graphs Improve Tabular Foundation Models?cited by 0
  34. 2026Tabular Incremental Inferencecited by 0
  35. 2026A Nationwide Japanese Medical Claims Foundation Model: Balancing Model Scaling and Task-Specific Computational Efficiencycited by 0
  36. 2026Efficient and Interpretable Transformer for Counterfactual Fairnesscited by 0
  37. 2026Learning to Select Source-Traceable Evidence for Language-Model Prediction from Irregular Clinical Time Seriescited by 0
  38. 2026Segment-driven Structural Induction and Semantic Alignment for Heterogeneous Tabular Representationcited by 0

50 most cited of 56 All 56 (JSON)