# Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens

arXiv: https://arxiv.org/abs/2605.24417 · HTML: https://mlsanity.com/paper/2605.24417/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 23 May 2026.

## Builds on (most cited first, 23 of 23)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [TableBench: A Comprehensive and Complex Benchmark for Table Question Answering](https://mlsanity.com/paper/2408.09174/) (2024)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [Large Scale Transfer Learning for Tabular Data via Language Modeling](https://mlsanity.com/paper/2406.12031/) (2024)
- [Scikit-dimension: a Python package for intrinsic dimension estimation](https://mlsanity.com/paper/2109.02596/) (2021)
- [DANets: Deep Abstract Networks for Tabular Data Classification and Regression](https://mlsanity.com/paper/2112.02962/) (2021)
- [TABLET: Learning From Instructions For Tabular Data](https://mlsanity.com/paper/2304.13188/) (2023)
- [Language models are weak learners](https://mlsanity.com/paper/2306.14101/) (2023)
- [Fractional norms and quasinorms do not help to overcome the curse of dimensionality](https://mlsanity.com/paper/2004.14230/) (2020)
- [Towards Better Serialization of Tabular Data for Few-shot Classification with Large Language Models](https://mlsanity.com/paper/2312.12464/) (2023)
- ["Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models](https://mlsanity.com/paper/2409.18594/) (2024)
- [When Large Language Models Know the Table: A Framework for Assessing Data Contamination in Tabular Datasets](https://mlsanity.com/paper/2510.20351/) (2025)
- [TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesis](https://mlsanity.com/paper/2602.02523/) (2026)

## Abstract (arXiv)

Every tabular classification task requires sufficient labeled data. When labels are scarce, practitioners can collect more examples, encode domain expertise into features, or accept lower performance. Large language models offer an alternative by leveraging knowledge acquired during pretraining. This raises three questions: (i) can zero-shot inference reduce the need for labeled data; (ii) can a richer prompt reduce the need for more examples; and (iii) can encoded domain knowledge compensate for missing data? To answer each under explicit, testable conditions, we introduce TabLens, a benchmark of 91 real-world and synthetic datasets across eight domains, plus eight industrial datasets with free-text columns. First, below a complexity threshold estimable from a dataset's intrinsic dimensionality before running an LLM, zero-shot inference matches the accuracy of up to about 16 labeled examples. This holds whether the model generates an answer or, more cheaply, scores candidate labels by log-probabilities, which loses almost no accuracy except for the smallest models. Above the threshold, the advantage disappears. On datasets with free-text columns, however, zero-shot LLMs outperform 16-shot TabPFN-3 Plus without manual feature engineering. Second, a richer prompt reduces the number of required demonstrations only when it includes both real column names and a plain-language task description; providing only one can be worse than neither, particularly at higher shot counts. Third, on LLM-synthetic datasets, explicit if--then expert rules help only when the prompt provides enough task context to interpret them, and even then, they substitute for at most about 16 real examples. Together, these results establish testable conditions under which LLMs can reduce data-collection and feature-engineering requirements for tabular classification.