# Language models are weak learners

arXiv: https://arxiv.org/abs/2306.14101 · HTML: https://mlsanity.com/paper/2306.14101/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 48 references that are in the map.
- First posted on arXiv 25 Jun 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://mlsanity.com/paper/2108.02035/) (2021)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)

## Built on by (most cited first, 22 of 22)

- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective](https://mlsanity.com/paper/2406.00793/) (2024)
- [Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning](https://mlsanity.com/paper/2406.08527/) (2024)
- [Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2403.19103/) (2024)
- [Tabular Transfer Learning via Prompting LLMs](https://mlsanity.com/paper/2408.11063/) (2024)
- [LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization](https://mlsanity.com/paper/2502.10648/) (2025)
- [ED-Copilot: Reduce Emergency Department Wait Time with Language Model Diagnostic Assistance](https://mlsanity.com/paper/2402.13448/) (2024)
- [LLM Meeting Decision Trees on Tabular Data](https://mlsanity.com/paper/2505.17918/) (2025)
- [Learning a Decision Tree Algorithm with Transformers](https://mlsanity.com/paper/2402.03774/) (2024)
- [Focus On This, Not That! Steering LLMs with Adaptive Feature Specification](https://mlsanity.com/paper/2410.22944/) (2024)
- [Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard](https://mlsanity.com/paper/2505.13421/) (2025)
- [On the Performance of LLMs for Real Estate Appraisal](https://mlsanity.com/paper/2506.11812/) (2025)
- [LLM Empowered Prototype Learning for Zero and Few-Shot Tasks on Tabular Data](https://mlsanity.com/paper/2508.09263/) (2025)
- [Agentic Systems as Boosting Weak Reasoning Models](https://mlsanity.com/paper/2605.14163/) (2026)
- [Talking Trees: Reasoning-Assisted Induction of Decision Trees for Tabular Data](https://mlsanity.com/paper/2509.21465/) (2025)
- [LML-DAP: Language Model Learning a Dataset for Data-Augmented Prediction](https://mlsanity.com/paper/2409.18957/) (2024)
- [PENEX: AdaBoost-Inspired Neural Network Regularization](https://mlsanity.com/paper/2510.02107/) (2025)
- [What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization](https://mlsanity.com/paper/2601.17609/) (2026)
- [LLM Sparsity Prior for Robust Feature Selection](https://mlsanity.com/paper/2605.23102/) (2026)
- [Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens](https://mlsanity.com/paper/2605.24417/) (2026)
- [Coding LLMs Can Be Harnessed to Learn Interpretable Models from Data](https://mlsanity.com/paper/2610.14006/) (2026)

## Abstract (arXiv)

A central notion in practical and theoretical machine learning is that of a textitweak learner, classifiers that achieve better-than-random performance (on any given distribution over data), even by a small margin. Such weak learners form the practical basis for canonical machine learning methods such as boosting. In this work, we illustrate that prompt-based large language models can operate effectively as said weak learners. Specifically, we illustrate the use of a large language model (LLM) as a weak learner in a boosting algorithm applied to tabular data. We show that by providing (properly sampled according to the distribution of interest) text descriptions of tabular data samples, LLMs can produce a summary of the samples that serves as a template for classification and achieves the aim of acting as a weak learner on this task. We incorporate these models into a boosting approach, which in some settings can leverage the knowledge within the LLM to outperform traditional tree-based boosting. The model outperforms both few-shot learning and occasionally even more involved fine-tuning procedures, particularly for tasks involving small numbers of data points. The results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.