# Transfer Learning of Tabular Data by Finetuning Large Language Models

arXiv: https://arxiv.org/abs/2501.06863 · HTML: https://mlsanity.com/paper/2501.06863/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 12 Jan 2025.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records](https://mlsanity.com/paper/2401.07128/) (2024)
- [STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables](https://mlsanity.com/paper/2303.00918/) (2023)
- [T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interaction](https://mlsanity.com/paper/2211.16887/) (2022)
- [Survey on Deep Neural Networks in Speech and Vision Systems](https://mlsanity.com/paper/1908.07656/) (2019)
- [TABLET: Learning From Instructions For Tabular Data](https://mlsanity.com/paper/2304.13188/) (2023)
- [The Health Gym: Synthetic Health-Related Datasets for the Development of Reinforcement Learning Algorithms](https://mlsanity.com/paper/2203.06369/) (2022)
- [Deep Clustering of Tabular Data by Weighted Gaussian Distribution Learning](https://mlsanity.com/paper/2301.00802/) (2023)
- [Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking](https://mlsanity.com/paper/2401.04266/) (2024)
- [Effectiveness of Deep Image Embedding Clustering Methods on Tabular Data](https://mlsanity.com/paper/2212.14111/) (2022)
- [Gradient Boosting Trees and Large Language Models for Tabular Data Few-Shot Learning](https://mlsanity.com/paper/2411.04324/) (2024)

## Built on by (most cited first, 4 of 4)

- [LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains](https://mlsanity.com/paper/2511.06161/) (2025)
- [Deep Learning with Pretrained 'Internal World' Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction](https://mlsanity.com/paper/2504.18562/) (2025)
- [Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data](https://mlsanity.com/paper/2608.28209/) (2026)
- [When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction](https://mlsanity.com/paper/2608.30086/) (2026)

## Abstract (arXiv)

Despite the artificial intelligence (AI) revolution, deep learning has yet to achieve much success with tabular data due to heterogeneous feature space and limited sample sizes without viable transfer learning. The new era of generative AI, powered by large language models (LLM), brings unprecedented learning opportunities to diverse data and domains. This paper investigates the effectiveness of an LLM application programming interface (API) and transfer learning of LLM in tabular data classification. LLM APIs respond to input text prompts with tokenized data and instructions, whereas transfer learning finetunes an LLM for a target classification task. This paper proposes an end-to-end finetuning of LLM to demonstrate cross-data transfer learning on ten benchmark data sets when large pre-trained tabular data models do not exist to facilitate transfer learning. The proposed LLM finetuning method outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features - a standard feature size for tabular data sets. The transfer learning approach uses a fraction of the computational cost of other deep learning or API-based solutions while ensuring competitive or superior classification performance.