mlsanityData to 7 Oct 2026

"Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models

arXiv 2409.18594 ↗First posted 27 Sep 2024cs.AI cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) provide powerful means to leverage prior knowledge for predictive modeling when data is limited. In this work, we demonstrate how LLMs can use their compressed world knowledge to generate intrinsically interpretable machine learning models, i.e., decision trees, without any training data. We find that these zero-shot decision trees can even surpass data-driven trees on some small-sized tabular datasets and that embeddings derived from these trees perform better than data-driven tree-based embeddings on average. Our decision tree induction and embedding approaches can therefore serve as new knowledge-driven baselines for data-driven machine learning methods in the low-data regime. Furthermore, they offer ways to harness the rich world knowledge within LLMs for tabular machine learning tasks. Our code and results are available at https://github.com/ml-lab-htw/llm-trees.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 115 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

35 of 115 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2022A Survey on In-context Learningcited by 717
  11. 2022Transformers learn in-context by gradient descentcited by 714
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
23 more
  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  2. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 368
  3. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  4. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  5. 2024Capabilities of Gemini Models in Medicinecited by 231
  6. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 218
  7. 2017PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparisoncited by 139
  8. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  9. 2015Should we really use post-hoc tests based on mean-ranks?cited by 116
  10. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 90
  11. 2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 79
  12. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoningcited by 48
  13. 2024Is Mamba Capable of In-Context Learning?cited by 45
  14. 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 32
  15. 2018AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learningcited by 22
  16. 2024Elephants Never Forget: Testing Language Models for Memorization of Tabular Datacited by 19
  17. 2017Sparse Classification: a scalable discrete optimization perspectivecited by 18
  18. 2024Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarkscited by 17
  19. 2022AutoPrognosis 2.0: Democratizing Diagnostic and Prognostic Modeling in Healthcare with Automated Machine Learningcited by 10
  20. 2024Large Language Models are Interpretable Learnerscited by 7
  21. 2024PMLBmini: A Tabular Classification Benchmark Suite for Data-Scarce Applicationscited by 5
  22. 2023Cost-Sensitive Best Subset Selection for Logistic Regression: A Mixed-Integer Conic Optimization Perspectivecited by 3
  23. 2024Squeezing Lemons with Hammers: An Evaluation of AutoML and Tabular Deep Learning for Data-Scarce Classification Applicationscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Ensembling LLM-Induced Decision Trees for Explainable and Robust Error Detectioncited by 2
  2. 2025LogiPart: Local Large Language Models for Data Exploration at Scale with Logical Partitioningcited by 0
  3. 2026Embedding World Knowledge into Tabular Models: Towards Best Practices for Embedding Pipeline Designcited by 0
  4. 2026Automated Instruction Revision (AIR): A Structured Comparison of Task Adaptation Strategies for LLMcited by 0
  5. 2026LLMTabBench: Evaluating LLMs on Binary Tabular Classification From Zero to Few Shotscited by 0
  6. 2026Marginal Response Surface Elicitation for Zero-Label Tabular Learningcited by 0