# Evolutionary Large Language Model for Automated Feature Transformation

arXiv: https://arxiv.org/abs/2405.16203 · HTML: https://mlsanity.com/paper/2405.16203/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 25 May 2024.

## Builds on (most cited first, 16 of 16)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://mlsanity.com/paper/2306.15626/) (2023)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation](https://mlsanity.com/paper/2307.03987/) (2023)
- [Language Model Crossover: Variation through Few-Shot Prompting](https://mlsanity.com/paper/2302.12170/) (2023)
- [Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap](https://mlsanity.com/paper/2401.10034/) (2024)
- [Feature Engineering for Predictive Modeling using Reinforcement Learning](https://mlsanity.com/paper/1709.07150/) (2017)
- [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://mlsanity.com/paper/2303.08518/) (2023)
- [The autofeat Python Library for Automated Feature Engineering and Selection](https://mlsanity.com/paper/1901.07329/) (2019)
- [Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction](https://mlsanity.com/paper/2205.14526/) (2022)
- [Teaching Language Models to Hallucinate Less with Synthetic Tasks](https://mlsanity.com/paper/2310.06827/) (2023)
- [Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossing](https://mlsanity.com/paper/2309.04612/) (2023)
- [Traceable Automatic Feature Transformation via Cascading Actor-Critic Agents](https://mlsanity.com/paper/2212.13402/) (2022)
- [Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions](https://mlsanity.com/paper/2309.13618/) (2023)

## Built on by (most cited first, 29 of 29)

- [MixLLM: Dynamic Routing in Mixed Large Language Models](https://mlsanity.com/paper/2502.18482/) (2025)
- [Entropy-based Exploration Conduction for Multi-step Reasoning](https://mlsanity.com/paper/2503.15848/) (2025)
- [Topology-aware Reinforcement Feature Space Reconstruction for Graph Data](https://mlsanity.com/paper/2411.05742/) (2024)
- [Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming](https://mlsanity.com/paper/2504.21304/) (2025)
- [Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives](https://mlsanity.com/paper/2506.24124/) (2025)
- [LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression](https://mlsanity.com/paper/2505.18602/) (2025)
- [Autonomous Data Agents: A New Opportunity for Smart Data](https://mlsanity.com/paper/2509.18710/) (2025)
- [LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformation](https://mlsanity.com/paper/2506.09085/) (2025)
- [Large Language Models for Extrapolative Modeling of Manufacturing Processes](https://mlsanity.com/paper/2502.12185/) (2025)
- [Bridging the Domain Gap in Equation Distillation with Reinforcement Feedback](https://mlsanity.com/paper/2505.15572/) (2025)
- [Distribution Shift Aware Neural Tabular Learning](https://mlsanity.com/paper/2508.19486/) (2025)
- [DELTA: Variational Disentangled Learning for Privacy-Preserving Data Reprogramming](https://mlsanity.com/paper/2509.00693/) (2025)
- [Retrieval-Augmented Feature Generation for Domain-Specific Classification](https://mlsanity.com/paper/2406.11177/) (2024)
- [ELATE: Evolutionary Language model for Automated Time-series Engineering](https://mlsanity.com/paper/2508.14667/) (2025)
- [Causally-Guided Automated Feature Engineering with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.16435/) (2026)
- [FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Data](https://mlsanity.com/paper/2510.25223/) (2025)
- [Building a Chinese Medical Dialogue System: Integrating Large-scale Corpora and Novel Models](https://mlsanity.com/paper/2410.03521/) (2024)
- [StorageXTuner: An LLM Agent-Driven Automatic Tuning Framework for Heterogeneous Storage Systems](https://mlsanity.com/paper/2510.25017/) (2025)
- [Dataforge: Agentic Platform for Autonomous Data Engineering](https://mlsanity.com/paper/2511.06185/) (2025)
- [DeepFeature: LLM-Empowered Context-aware Feature Generation for Wearable Biosignals](https://mlsanity.com/paper/2512.08379/) (2025)
- [The Semantic Architect: How FEAML Bridges Structured Data and LLMs for Multi-Label Tasks](https://mlsanity.com/paper/2512.15082/) (2025)
- [Multi-Agent Procedural Graph Extraction with Structural and Logical Refinement](https://mlsanity.com/paper/2601.19170/) (2026)
- [CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization](https://mlsanity.com/paper/2602.09851/) (2026)
- [Evolving Demonstration Optimization for Chain-of-Thought Feature Transformation](https://mlsanity.com/paper/2603.09987/) (2026)
- [EvoForest: A Novel Machine-Learning Paradigm via Open-Ended Evolution of Computational Graphs](https://mlsanity.com/paper/2604.19761/) (2026)
- [Deconstructing Spatial Complexity: Hierarchical Decomposition for LLM Spatial Reasoning](https://mlsanity.com/paper/2605.28144/) (2026)
- [LATTEArena: An Evaluation Framework for LLM-powered Tabular Feature Engineering (Extended Version)](https://mlsanity.com/paper/2606.09004/) (2026)
- [Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs](https://mlsanity.com/paper/2607.23286/) (2026)
- [Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses](https://mlsanity.com/paper/2609.35255/) (2026)

## Abstract (arXiv)

Feature transformation aims to reconstruct the feature space of raw features to enhance the performance of downstream models. However, the exponential growth in the combinations of features and operations poses a challenge, making it difficult for existing methods to efficiently explore a wide space. Additionally, their optimization is solely driven by the accuracy of downstream models in specific domains, neglecting the acquisition of general feature knowledge. To fill this research gap, we propose an evolutionary LLM framework for automated feature transformation. This framework consists of two parts: 1) constructing a multi-population database through an RL data collector while utilizing evolutionary algorithm strategies for database maintenance, and 2) utilizing the ability of Large Language Model (LLM) in sequence understanding, we employ few-shot prompts to guide LLM in generating superior samples based on feature transformation sequence distinction. Leveraging the multi-population database initially provides a wide search scope to discover excellent populations. Through culling and evolution, the high-quality populations are afforded greater opportunities, thereby furthering the pursuit of optimal individuals. Through the integration of LLMs with evolutionary algorithms, we achieve efficient exploration within a vast space, while harnessing feature knowledge to propel optimization, thus realizing a more adaptable search paradigm. Finally, we empirically demonstrate the effectiveness and generality of our proposed method.