# Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning

arXiv: https://arxiv.org/abs/2406.08527 · HTML: https://mlsanity.com/paper/2406.08527/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Reinforcement Learning for Combinatorial Optimization: A Survey](https://mlsanity.com/paper/2003.03600/) (2020)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [The autofeat Python Library for Automated Feature Engineering and Selection](https://mlsanity.com/paper/1901.07329/) (2019)
- [Using Large Language Models for Hyperparameter Optimization](https://mlsanity.com/paper/2312.04528/) (2023)
- [Making Pre-trained Language Models Great on Tabular Prediction](https://mlsanity.com/paper/2403.01841/) (2024)
- [OpenFE: Automated Feature Generation with Expert-level Performance](https://mlsanity.com/paper/2211.12507/) (2022)
- [STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables](https://mlsanity.com/paper/2303.00918/) (2023)
- [Trompt: Towards a Better Deep Neural Network for Tabular Data](https://mlsanity.com/paper/2305.18446/) (2023)

## Built on by (most cited first, 30 of 49)

- [MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement](https://mlsanity.com/paper/2506.15692/) (2025)
- [LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers](https://mlsanity.com/paper/2503.14434/) (2025)
- [A Comprehensive Survey of Synthetic Tabular Data Generation](https://mlsanity.com/paper/2504.16506/) (2025)
- [Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey](https://mlsanity.com/paper/2411.10478/) (2024)
- [Large Language Models Engineer Too Many Simple Features For Tabular Data](https://mlsanity.com/paper/2410.17787/) (2024)
- [LLM Meeting Decision Trees on Tabular Data](https://mlsanity.com/paper/2505.17918/) (2025)
- ["Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models](https://mlsanity.com/paper/2409.18594/) (2024)
- [Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers](https://mlsanity.com/paper/2411.01645/) (2024)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes](https://mlsanity.com/paper/2502.02672/) (2025)
- [FAMOSE: A ReAct Approach to Automated Feature Discovery](https://mlsanity.com/paper/2602.17641/) (2026)
- [Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard](https://mlsanity.com/paper/2505.13421/) (2025)
- [Tabular Feature Discovery With Reasoning Type Exploration](https://mlsanity.com/paper/2506.20357/) (2025)
- [Applications of Large Language Model Reasoning in Feature Generation](https://mlsanity.com/paper/2503.11989/) (2025)
- [Research Challenges in Relational Database Management Systems for LLM Queries](https://mlsanity.com/paper/2508.20912/) (2025)
- [FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Data](https://mlsanity.com/paper/2510.25223/) (2025)
- [Knowledge-Informed Automatic Feature Extraction via Collaborative Large Language Model Agents](https://mlsanity.com/paper/2511.15074/) (2025)
- [RelAgent: LLM Agents as Data Scientists for Relational Learning](https://mlsanity.com/paper/2605.07840/) (2026)
- [Talking Trees: Reasoning-Assisted Induction of Decision Trees for Tabular Data](https://mlsanity.com/paper/2509.21465/) (2025)
- [LLM as an Algorithmist: Enhancing Anomaly Detectors via Programmatic Synthesis](https://mlsanity.com/paper/2510.03904/) (2025)
- [NSR-Boost: A Neuro-Symbolic Residual Boosting Framework for Industrial Legacy Models](https://mlsanity.com/paper/2601.10457/) (2026)
- [Human-LLM Collaborative Feature Engineering for Tabular Data](https://mlsanity.com/paper/2601.21060/) (2026)
- [Automatic Construction of Clinical Scoring Systems with LLM Agents](https://mlsanity.com/paper/2601.22324/) (2026)
- [Interpretable DNA Sequence Classification via Dynamic Feature Generation in Decision Trees](https://mlsanity.com/paper/2604.12060/) (2026)
- [FeatEHR-LLM: Leveraging Large Language Models for Feature Engineering in Electronic Health Records](https://mlsanity.com/paper/2604.22534/) (2026)
- [BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classification](https://mlsanity.com/paper/2605.06117/) (2026)
- [Eureka: Intelligent Feature Engineering for Enterprise AI Cloud Resource Demand Prediction](https://mlsanity.com/paper/2605.25297/) (2026)
- [LLMs as Feature Engineers for Text-and-Tabular Prediction](https://mlsanity.com/paper/2609.21894/) (2026)
- [Flat-Consensus Diffusion for Robust Data Reshaping under Noisy Evaluator](https://mlsanity.com/paper/2609.32696/) (2026)
- [SIFOTL: A Principled, Statistically-Informed Fidelity-Optimization Method for Tabular Learning](https://mlsanity.com/paper/2507.17979/) (2025)

## Abstract (arXiv)

In tabular prediction tasks, tree-based models combined with automated feature engineering methods often outperform deep learning approaches that rely on learned representations. While these feature engineering techniques are effective, they typically depend on a pre-defined search space and primarily use validation scores for feature selection, thereby missing valuable insights from previous experiments. To address these limitations, we propose a novel tabular learning framework that utilizes large language models (LLMs), termed Optimizing Column feature generator with decision Tree reasoning (OCTree). Our key idea is to leverage the reasoning capabilities of LLMs to identify effective feature generation rules without manually specifying the search space and provide language-based reasoning information highlighting past experiments as feedback for iterative rule improvements. We use decision trees to convey this reasoning information, as they can be easily represented in natural language, effectively providing knowledge from prior experiments (i.e., the impact of the generated features on performance) to the LLMs. Our empirical results demonstrate that OCTree consistently enhances the performance of various prediction models across diverse benchmarks, outperforming competing automated feature engineering methods. Code is available at https://github.com/jaehyun513/OCTree.