# ELF-Gym: Evaluating Large Language Models Generated Features for Tabular Prediction

arXiv: https://arxiv.org/abs/2410.12865 · HTML: https://mlsanity.com/paper/2410.12865/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 13 Oct 2024.

## Builds on (most cited first, 12 of 12)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Shortcut Learning of Large Language Models in Natural Language Understanding](https://mlsanity.com/paper/2208.11857/) (2022)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [The autofeat Python Library for Automated Feature Engineering and Selection](https://mlsanity.com/paper/1901.07329/) (2019)
- [Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models](https://mlsanity.com/paper/2310.10449/) (2023)
- [Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints](https://mlsanity.com/paper/2302.09185/) (2023)
- [SAFE: Scalable Automatic Feature Engineering Framework for Industrial Tasks](https://mlsanity.com/paper/2003.02556/) (2020)

## Built on by (most cited first, 10 of 10)

- [Large Language Models Engineer Too Many Simple Features For Tabular Data](https://mlsanity.com/paper/2410.17787/) (2024)
- [No Need to Train Your RDB Foundation Model](https://mlsanity.com/paper/2602.13697/) (2026)
- [FAMOSE: A ReAct Approach to Automated Feature Discovery](https://mlsanity.com/paper/2602.17641/) (2026)
- [Language Model Representations for Efficient Few-Shot Tabular Classification](https://mlsanity.com/paper/2602.15844/) (2026)
- [Multi-level Diagnosis and Evaluation for Robust Tabular Feature Engineering with Large Language Models](https://mlsanity.com/paper/2509.25207/) (2025)
- [ConCISE: A Reference-Free Conciseness Evaluation Metric for LLM-Generated Answers](https://mlsanity.com/paper/2511.16846/) (2025)
- [Human-LLM Collaborative Feature Engineering for Tabular Data](https://mlsanity.com/paper/2601.21060/) (2026)
- [Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data](https://mlsanity.com/paper/2604.20261/) (2026)
- [InvWeaver: Deductive Feedback for Invariant Synthesis in Interacting-Loop Programs](https://mlsanity.com/paper/2607.05478/) (2026)
- [SIGMA: SHAP-Guided Implicit-Trajectory Generation for Metadata-Free LLM-Based AutoFE](https://mlsanity.com/paper/2608.17948/) (2026)

## Abstract (arXiv)

Crafting effective features is a crucial yet labor-intensive and domain-specific task within machine learning pipelines. Fortunately, recent advancements in Large Language Models (LLMs) have shown promise in automating various data science tasks, including feature engineering. But despite this potential, evaluations thus far are primarily based on the end performance of a complete ML pipeline, providing limited insight into precisely how LLMs behave relative to human experts in feature engineering. To address this gap, we propose ELF-Gym, a framework for Evaluating LLM-generated Features. We curated a new dataset from historical Kaggle competitions, including 251 "golden" features used by top-performing teams. ELF-Gym then quantitatively evaluates LLM-generated features by measuring their impact on downstream model performance as well as their alignment with expert-crafted features through semantic and functional similarity assessments. This approach provides a more comprehensive evaluation of disparities between LLMs and human experts, while offering valuable insights into specific areas where LLMs may have room for improvement. For example, using ELF-Gym we empirically demonstrate that, in the best-case scenario, LLMs can semantically capture approximately 56% of the golden features, but at the more demanding implementation level this overlap drops to 13%. Moreover, in other cases LLMs may fail completely, particularly on datasets that require complex features, indicating broad potential pathways for improvement.