mlsanityData to 14 Oct 2026

Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning

arXiv 2406.08527 ↗First posted 12 Jun 2024cs.LG cs.AIcited by 49

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

In tabular prediction tasks, tree-based models combined with automated feature engineering methods often outperform deep learning approaches that rely on learned representations. While these feature engineering techniques are effective, they typically depend on a pre-defined search space and primarily use validation scores for feature selection, thereby missing valuable insights from previous experiments. To address these limitations, we propose a novel tabular learning framework that utilizes large language models (LLMs), termed Optimizing Column feature generator with decision Tree reasoning (OCTree). Our key idea is to leverage the reasoning capabilities of LLMs to identify effective feature generation rules without manually specifying the search space and provide language-based reasoning information highlighting past experiments as feedback for iterative rule improvements. We use decision trees to convey this reasoning information, as they can be easily represented in natural language, effectively providing knowledge from prior experiments (i.e., the impact of the generated features on performance) to the LLMs. Our empirical results demonstrate that OCTree consistently enhances the performance of various prediction models across diverse benchmarks, outperforming competing automated feature engineering methods. Code is available at https://github.com/jaehyun513/OCTree.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 49 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

35 of 49 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  6. 2014Going Deeper with Convolutionscited by 4,241
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2019Contrastive Multiview Codingcited by 1,301
  11. 2023Large Language Models as Optimizerscited by 864
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
23 more
  1. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  2. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  3. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  4. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  5. 2020Reinforcement Learning for Combinatorial Optimization: A Surveycited by 236
  6. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  7. 2018An efficient framework for learning sentence representationscited by 198
  8. 2017OpenML Benchmarking Suitescited by 178
  9. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  10. 2023Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineeringcited by 127
  11. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 104
  12. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  13. 2019The autofeat Python Library for Automated Feature Engineering and Selectioncited by 73
  14. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  15. 2024Making Pre-trained Language Models Great on Tabular Predictioncited by 57
  16. 2022OpenFE: Automated Feature Generation with Expert-level Performancecited by 52
  17. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 49
  18. 2023Trompt: Towards a Better Deep Neural Network for Tabular Datacited by 36
  19. 2023A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learningcited by 24
  20. 2023Language models are weak learnerscited by 22
  21. 2024HyperFast: Instant Classification for Tabular Datacited by 22
  22. 2020A Survey on Recent Progress in the Theory of Evolutionary Algorithms for Discrete Optimizationcited by 4
  23. 2024Automated Model Selection for Tabular Datacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 27.

49 papers.

Sort:

By area:

  1. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 58
  2. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizerscited by 38
  3. 2025A Comprehensive Survey of Synthetic Tabular Data Generationcited by 28
  4. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  5. 2024Large Language Models Engineer Too Many Simple Features For Tabular Datacited by 10
  6. 2025LLM Meeting Decision Trees on Tabular Datacited by 8
  7. 2024"Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Modelscited by 6
  8. 2024Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifierscited by 6
  9. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  10. 2025Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizescited by 4
  11. 2026FAMOSE: A ReAct Approach to Automated Feature Discoverycited by 4
  12. 2025Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboardcited by 3
37 more
  1. 2025Tabular Feature Discovery With Reasoning Type Explorationcited by 3
  2. 2025Applications of Large Language Model Reasoning in Feature Generationcited by 2
  3. 2025Research Challenges in Relational Database Management Systems for LLM Queriescited by 2
  4. 2025FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Datacited by 2
  5. 2025Knowledge-Informed Automatic Feature Extraction via Collaborative Large Language Model Agentscited by 2
  6. 2026RelAgent: LLM Agents as Data Scientists for Relational Learningcited by 2
  7. 2025Talking Trees: Reasoning-Assisted Induction of Decision Trees for Tabular Datacited by 1
  8. 2025LLM as an Algorithmist: Enhancing Anomaly Detectors via Programmatic Synthesiscited by 1
  9. 2026NSR-Boost: A Neuro-Symbolic Residual Boosting Framework for Industrial Legacy Modelscited by 1
  10. 2026Human-LLM Collaborative Feature Engineering for Tabular Datacited by 1
  11. 2026Automatic Construction of Clinical Scoring Systems with LLM Agentscited by 1
  12. 2026Interpretable DNA Sequence Classification via Dynamic Feature Generation in Decision Treescited by 1
  13. 2026FeatEHR-LLM: Leveraging Large Language Models for Feature Engineering in Electronic Health Recordscited by 1
  14. 2026BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classificationcited by 1
  15. 2026Eureka: Intelligent Feature Engineering for Enterprise AI Cloud Resource Demand Predictioncited by 1
  16. 2026LLMs as Feature Engineers for Text-and-Tabular Predictioncited by 1
  17. 2026Flat-Consensus Diffusion for Robust Data Reshaping under Noisy Evaluatorcited by 1
  18. 2025SIFOTL: A Principled, Statistically-Informed Fidelity-Optimization Method for Tabular Learningcited by 0
  19. 2025DeepFeature: LLM-Empowered Context-aware Feature Generation for Wearable Biosignalscited by 0
  20. 2025The Semantic Architect: How FEAML Bridges Structured Data and LLMs for Multi-Label Taskscited by 0
  21. 2025From Coverage to Causes: Data-Centric Fuzzing for JavaScript Enginescited by 0
  22. 2025Exploring the Heterogeneity of Tabular Data: A Diversity-aware Data Generator via LLMscited by 0
  23. 2026CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimizationcited by 0
  24. 2026Evolutionary Router Feature Generation for Zero-Shot Graph Anomaly Detection with Mixture-of-Expertscited by 0
  25. 2026iML: Executable, Problem-Grounded, and Broadly Exploratory Code-Driven AutoMLcited by 0
  26. 2026MedFeat: Model-Aware and Explainability-Driven Feature Engineering with LLMs for Tabular Predictioncited by 0
  27. 2026Adaptive Discovery of Interpretable Audio Attributes with Multimodal LLMs for Low-Resource Classificationcited by 0
  28. 2026Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Datacited by 0
  29. 2026Concordia: Self-Improving Synthetic Tables for Federated LLMscited by 0
  30. 2026Learning to Select Source-Traceable Evidence for Language-Model Prediction from Irregular Clinical Time Seriescited by 0
  31. 2026LATTEArena: An Evaluation Framework for LLM-powered Tabular Feature Engineering (Extended Version)cited by 0
  32. 2026Evolutionary Feature Engineering for Structured Datacited by 0
  33. 2026Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMscited by 0
  34. 2026When Do Corrective Features Help? An Agent for Corrective Feature Discovery on Black-Box Forecasterscited by 0
  35. 2026SIGMA: SHAP-Guided Implicit-Trajectory Generation for Metadata-Free LLM-Based AutoFEcited by 0
  36. 2026SymboLLM-FE: LLM-Accelerated Symbolic Regression for Automated Feature Engineering on Tabular Datacited by 0
  37. 2026TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Prompts to Trees: Effective LLM-Guided Tree Generation for Few-Shot Tabular Classification
  2. 2026Feature Generation Using LLMs: An Evolutionary Algorithm Approach
  3. 2025Strengthening LLMs for Tabular Prediction with Structural Priors
  4. 2025Utilizing Training Data to Improve LLM Reasoning for Tabular Understanding
  5. 2026TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks
  6. 2026ReSS: Learning Reasoning Models for Tabular Data Prediction via Symbolic Scaffold
  7. 2026ProgramTab: Boosting Table Reasoning of LLMs via Programmatic Paradigm
  8. 2024Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understanding