# CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2602.09851 · HTML: https://mlsanity.com/paper/2602.09851/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 63 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Builds on (most cited first, 30 of 33)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [AutoML-GPT: Automatic Machine Learning with GPT](https://mlsanity.com/paper/2305.02499/) (2023)
- [Feature Engineering for Predictive Modeling using Reinforcement Learning](https://mlsanity.com/paper/1709.07150/) (2017)
- [The autofeat Python Library for Automated Feature Engineering and Selection](https://mlsanity.com/paper/1901.07329/) (2019)
- [AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions](https://mlsanity.com/paper/2410.20424/) (2024)
- [Large Language Model Agent for Hyper-Parameter Optimization](https://mlsanity.com/paper/2402.01881/) (2024)
- [ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoning](https://mlsanity.com/paper/2506.16499/) (2025)
- [LMPriors: Pre-Trained Language Models as Task-Specific Priors](https://mlsanity.com/paper/2210.12530/) (2022)
- [OpenFE: Automated Feature Generation with Expert-level Performance](https://mlsanity.com/paper/2211.12507/) (2022)
- [Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning](https://mlsanity.com/paper/2406.08527/) (2024)
- [An ADMM Based Framework for AutoML Pipeline Configuration](https://mlsanity.com/paper/1905.00424/) (2019)
- [LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers](https://mlsanity.com/paper/2503.14434/) (2025)
- [LightAutoML: AutoML Solution for a Large Financial Services Ecosystem](https://mlsanity.com/paper/2109.01528/) (2021)
- [Efficient Automatic CASH via Rising Bandits](https://mlsanity.com/paper/2012.04371/) (2020)
- [Evolutionary Large Language Model for Automated Feature Transformation](https://mlsanity.com/paper/2405.16203/) (2024)
- [Dynamic and Adaptive Feature Generation with LLM](https://mlsanity.com/paper/2406.03505/) (2024)
- [AlphaD3M: Machine Learning Pipeline Synthesis](https://mlsanity.com/paper/2111.02508/) (2021)
- [Data Formulator 2: Iterative Creation of Data Visualizations, with AI Transforming Data Along the Way](https://mlsanity.com/paper/2408.16119/) (2024)
- [Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey](https://mlsanity.com/paper/2411.10478/) (2024)
- [MFES-HB: Efficient Hyperband with Multi-Fidelity Quality Measurements](https://mlsanity.com/paper/2012.03011/) (2020)
- [Language-Based Bayesian Optimization Research Assistant (BORA)](https://mlsanity.com/paper/2501.16224/) (2025)
- [Automated data processing and feature engineering for deep learning and big data applications: a survey](https://mlsanity.com/paper/2403.11395/) (2024)
- [AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models](https://mlsanity.com/paper/2408.00665/) (2024)

## Abstract (arXiv)

Feature Engineering (FE) is pivotal in automated machine learning (AutoML) but remains a bottleneck for traditional methods, which operate within rigid search spaces and lack domain awareness. While Large Language Models (LLMs) offer a promising alternative to generate unbounded operators with semantic reasoning, existing methods focus on isolated subtasks such as feature generation, falling short of free-form FE pipelines. Moreover, they are rarely coupled with hyperparameter optimization (HPO) of the downstream ML model, leading to greedy "FE-then-HPO" workflows that cannot capture strong FE-HPO interactions. In this paper, we present CoFEH, a collaborative framework that interleaves LLM-based FE and Bayesian HPO for robust end-to-end AutoML. CoFEH uses an LLM-driven FE optimizer powered by Tree of Thought (TOT) to explore flexible FE pipelines, a Bayesian optimization (BO) module to solve HPO, and a dynamic optimizer selector that adaptively interleaves FE and HPO steps. Crucially, we introduce a mutual conditioning mechanism that shares context between LLM and BO, enabling mutually informed decisions. Experiments show that CoFEH outperforms both traditional and LLM-based baselines in both standalone FE and joint FE+HPO settings.