# Dynamic and Adaptive Feature Generation with LLM

arXiv: https://arxiv.org/abs/2406.03505 · HTML: https://mlsanity.com/paper/2406.03505/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [Active Prompting with Chain-of-Thought for Large Language Models](https://mlsanity.com/paper/2302.12246/) (2023)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [Feature Engineering for Predictive Modeling using Reinforcement Learning](https://mlsanity.com/paper/1709.07150/) (2017)
- [Towards Better Chain-of-Thought Prompting Strategies: A Survey](https://mlsanity.com/paper/2310.04959/) (2023)
- [Group-wise Reinforcement Feature Generation for Optimal and Explainable Representation Space Reconstruction](https://mlsanity.com/paper/2205.14526/) (2022)
- [An Overview on Data Representation Learning: From Traditional Feature Learning to Recent Deep Learning](https://mlsanity.com/paper/1611.08331/) (2016)
- [Techniques for Automated Machine Learning](https://mlsanity.com/paper/1907.08908/) (2019)
- [Automated data processing and feature engineering for deep learning and big data applications: a survey](https://mlsanity.com/paper/2403.11395/) (2024)
- [TFWT: Tabular Feature Weighting with Transformer](https://mlsanity.com/paper/2405.08403/) (2024)

## Built on by (most cited first, 27 of 27)

- [Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey](https://mlsanity.com/paper/2411.10478/) (2024)
- [AR Overlay: Training Image Pose Estimation on Curved Surface in a Synthetic Way](https://mlsanity.com/paper/2409.14577/) (2024)
- [LLM-based feature generation from text for interpretable machine learning](https://mlsanity.com/paper/2409.07132/) (2024)
- [Topology-aware Reinforcement Feature Space Reconstruction for Graph Data](https://mlsanity.com/paper/2411.05742/) (2024)
- [Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming](https://mlsanity.com/paper/2504.21304/) (2025)
- [In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminates](https://mlsanity.com/paper/2410.15612/) (2024)
- [DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models](https://mlsanity.com/paper/2504.18053/) (2025)
- [FastFT: Accelerating Reinforced Feature Transformation via Advanced Exploration Strategies](https://mlsanity.com/paper/2503.20394/) (2025)
- [LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformation](https://mlsanity.com/paper/2506.09085/) (2025)
- [Automatic Prompt Optimization for Dataset-Level Feature Discovery](https://mlsanity.com/paper/2601.13922/) (2026)
- [Causally-Guided Automated Feature Engineering with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.16435/) (2026)
- [Enhancing Risk Assessment in Transformers with Loss-at-Risk Functions](https://mlsanity.com/paper/2411.02558/) (2024)
- [Towards Data-Centric AI: A Comprehensive Survey of Traditional, Reinforcement, and Generative Approaches for Tabular Data Transformation](https://mlsanity.com/paper/2501.10555/) (2025)
- [Biological Sequence with Language Model Prompting: A Survey](https://mlsanity.com/paper/2503.04135/) (2025)
- [Applications of Large Language Model Reasoning in Feature Generation](https://mlsanity.com/paper/2503.11989/) (2025)
- [Collaborative Multi-Agent Reinforcement Learning for Automated Feature Transformation with Graph-Driven Path Optimization](https://mlsanity.com/paper/2504.17355/) (2025)
- [LEKA:LLM-Enhanced Knowledge Augmentation](https://mlsanity.com/paper/2501.17802/) (2025)
- [Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision](https://mlsanity.com/paper/2502.15147/) (2025)
- [ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts](https://mlsanity.com/paper/2511.11883/) (2025)
- [Flat-Consensus Diffusion for Robust Data Reshaping under Noisy Evaluator](https://mlsanity.com/paper/2609.32696/) (2026)
- [DeepFeature: LLM-Empowered Context-aware Feature Generation for Wearable Biosignals](https://mlsanity.com/paper/2512.08379/) (2025)
- [From Coverage to Causes: Data-Centric Fuzzing for JavaScript Engines](https://mlsanity.com/paper/2512.18102/) (2025)
- [CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization](https://mlsanity.com/paper/2602.09851/) (2026)
- [Evolutionary Router Feature Generation for Zero-Shot Graph Anomaly Detection with Mixture-of-Experts](https://mlsanity.com/paper/2602.11622/) (2026)
- [LATTEArena: An Evaluation Framework for LLM-powered Tabular Feature Engineering (Extended Version)](https://mlsanity.com/paper/2606.09004/) (2026)
- [Feature Generation Using LLMs: An Evolutionary Algorithm Approach](https://mlsanity.com/paper/2607.16255/) (2026)
- [Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs](https://mlsanity.com/paper/2607.23286/) (2026)

## Abstract (arXiv)

The representation of feature space is a crucial environment where data points get vectorized and embedded for subsequent modeling. Thus the efficacy of machine learning (ML) algorithms is closely related to the quality of feature engineering. As one of the most important techniques, feature generation transforms raw data into an optimized feature space conducive to model training and further refines the space. Despite the advancements in automated feature engineering and feature generation, current methodologies often suffer from three fundamental issues: lack of explainability, limited applicability, and inflexible strategy. These shortcomings frequently hinder and limit the deployment of ML models across varied scenarios. Our research introduces a novel approach adopting large language models (LLMs) and feature-generating prompts to address these challenges. We propose a dynamic and adaptive feature generation method that enhances the interpretability of the feature generation process. Our approach broadens the applicability across various data types and tasks and offers advantages over strategic flexibility. A broad range of experiments showcases that our approach is significantly superior to existing methods.