# Decision Tree Induction Through LLMs via Semantically-Aware Evolution

arXiv: https://arxiv.org/abs/2503.14217 · HTML: https://mlsanity.com/paper/2503.14217/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 85 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution](https://mlsanity.com/paper/2402.01145/) (2024)
- [EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers](https://mlsanity.com/paper/2309.08532/) (2023)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Evolution through Large Models](https://mlsanity.com/paper/2206.08896/) (2022)
- [Language Model Crossover: Variation through Few-Shot Prompting](https://mlsanity.com/paper/2302.12170/) (2023)
- [Optimal Sparse Decision Trees](https://mlsanity.com/paper/1904.12847/) (2019)
- [Generalized and Scalable Optimal Sparse Decision Trees](https://mlsanity.com/paper/2006.08690/) (2020)
- [LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization](https://mlsanity.com/paper/2306.01102/) (2023)
- [Using Centroidal Voronoi Tessellations to Scale Up the Multi-dimensional Archive of Phenotypic Elites Algorithm](https://mlsanity.com/paper/1610.05729/) (2016)
- [Interpretable Policies for Reinforcement Learning by Genetic Programming](https://mlsanity.com/paper/1712.04170/) (2017)
- [Fully Autonomous Programming with Large Language Models](https://mlsanity.com/paper/2304.10423/) (2023)
- [Position: Leverage Foundational Models for Black-Box Optimization](https://mlsanity.com/paper/2405.03547/) (2024)
- [Enhancing Genetic Improvement Mutations Using Large Language Models](https://mlsanity.com/paper/2310.19813/) (2023)
- [GradTree: Learning Axis-Aligned Decision Trees with Gradient Descent](https://mlsanity.com/paper/2305.03515/) (2023)
- [MAPTree: Beating "Optimal" Decision Trees with Bayesian Decision Trees](https://mlsanity.com/paper/2309.15312/) (2023)
- [Evolutionary NAS with Gene Expression Programming of Cellular Encoding](https://mlsanity.com/paper/2005.13110/) (2020)

## Built on by (most cited first, 6 of 6)

- [LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression](https://mlsanity.com/paper/2505.18602/) (2025)
- [Timely Clinical Diagnosis through Active Test Selection](https://mlsanity.com/paper/2510.18988/) (2025)
- [Hypothesis Hunting with Evolving Networks of Autonomous Scientific Agents](https://mlsanity.com/paper/2510.08619/) (2025)
- [Automatic Construction of Clinical Scoring Systems with LLM Agents](https://mlsanity.com/paper/2601.22324/) (2026)
- [LogiPart: Local Large Language Models for Data Exploration at Scale with Logical Partitioning](https://mlsanity.com/paper/2509.22211/) (2025)
- [CAMI: Cost-Aware Agent-Guided Multi-Indexing for Semantic Retrieval](https://mlsanity.com/paper/2606.28365/) (2026)

## Abstract (arXiv)

Decision trees are a crucial class of models offering robust predictive performance and inherent interpretability across various domains, including healthcare, finance, and logistics. However, current tree induction methods often face limitations such as suboptimal solutions from greedy methods or prohibitive computational costs and limited applicability of exact optimization approaches. To address these challenges, we propose an evolutionary optimization method for decision tree induction based on genetic programming (GP). Our key innovation is the integration of semantic priors and domain-specific knowledge about the search space into the optimization algorithm. To this end, we introduce textttLLEGO, a framework that incorporates semantic priors into genetic search operators through the use of Large Language Models (LLMs), thereby enhancing search efficiency and targeting regions of the search space that yield decision trees with superior generalization performance. This is operationalized through novel genetic operators that work with structured natural language prompts, effectively utilizing LLMs as conditional generative models and sources of semantic knowledge. Specifically, we introduce textitfitness-guided crossover to exploit high-performing regions, and textitdiversity-guided mutation for efficient global exploration of the search space. These operators are controlled by corresponding hyperparameters that enable a more nuanced balance between exploration and exploitation across the search space. Empirically, we demonstrate across various benchmarks that textttLLEGO evolves superior-performing trees compared to existing tree induction methods, and exhibits significantly more efficient search performance compared to conventional GP approaches.