mlsanityData to 14 Oct 2026

Gradient Boosting Trees and Large Language Models for Tabular Data Few-Shot Learning

arXiv 2411.04324 ↗First posted 6 Nov 2024cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLM) have brought numerous of new applications to Machine Learning (ML). In the context of tabular data (TD), recent studies show that TabLLM is a very powerful mechanism for few-shot-learning (FSL) applications, even if gradient boosting decisions trees (GBDT) have historically dominated the TD field. In this work we demonstrate that although LLMs are a viable alternative, the evidence suggests that baselines used to gauge performance can be improved. We replicated public benchmarks and our methodology improves LightGBM by 290%, this is mainly driven by forcing node splitting with few samples, a critical step in FSL with GBDT. Our results show an advantage to TabLLM for 8 or fewer shots, but as the number of samples increases GBDT provides competitive performance at a fraction of runtime. For other real-life applications with vast number of samples, we found FSL still useful to improve model diversity, and when combined with ExtraTrees it provides strong resilience to overfitting, our proposal was validated in a ML competition setting ranking first place.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

33 of 49 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  10. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  11. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  12. 2023A Comprehensive Overview of Large Language Modelscited by 648
21 more
  1. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  3. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  4. 2017Deep & Cross Network for Ad Click Predictionscited by 393
  5. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
  6. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  7. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 199
  8. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  9. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  10. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  11. 2023Self-planning Code Generation with Large Language Modelscited by 124
  12. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 113
  13. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  14. 2022Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarioscited by 65
  15. 2021Boost then Convolve: Gradient Boosting Meets Graph Neural Networkscited by 44
  16. 2018An Interpretable Model with Globally Consistent Explanations for Credit Riskcited by 33
  17. 2017CatBoost: unbiased boosting with categorical featurescited by 31
  18. 2024Elephants Never Forget: Testing Language Models for Memorization of Tabular Datacited by 19
  19. 2018DeepFM: An End-to-End Wide & Deep Learning Framework for CTR Predictioncited by 9
  20. 2023UniPredict: Large Language Models are Universal Tabular Classifierscited by 9
  21. 2022DIWIFT: Discovering Instance-wise Influential Features for Tabular Datacited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Transfer Learning of Tabular Data by Finetuning Large Language Modelscited by 4
  2. 2025LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domainscited by 2
  3. 2026The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilancecited by 0
  4. 2026The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Eracited by 0
  5. 2026When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Predictioncited by 0