# Tabular Feature Discovery With Reasoning Type Exploration

arXiv: https://arxiv.org/abs/2506.20357 · HTML: https://mlsanity.com/paper/2506.20357/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 25 Jun 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 18 of 18)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [Feature Engineering for Predictive Modeling using Reinforcement Learning](https://mlsanity.com/paper/1709.07150/) (2017)
- [The autofeat Python Library for Automated Feature Engineering and Selection](https://mlsanity.com/paper/1901.07329/) (2019)
- [Algorithms for multi-armed bandit problems](https://mlsanity.com/paper/1402.6028/) (2014)
- [OpenFE: Automated Feature Generation with Expert-level Performance](https://mlsanity.com/paper/2211.12507/) (2022)
- [Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning](https://mlsanity.com/paper/2406.08527/) (2024)
- [AutoCross: Automatic Feature Crossing for Tabular Data in Real-World Applications](https://mlsanity.com/paper/1904.12857/) (2019)
- [LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers](https://mlsanity.com/paper/2503.14434/) (2025)
- [Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models](https://mlsanity.com/paper/2404.06209/) (2024)
- [SAFE: Scalable Automatic Feature Engineering Framework for Industrial Tasks](https://mlsanity.com/paper/2003.02556/) (2020)
- [Large Language Models Engineer Too Many Simple Features For Tabular Data](https://mlsanity.com/paper/2410.17787/) (2024)
- [Auto-FP: An Experimental Study of Automated Feature Preprocessing for Tabular Data](https://mlsanity.com/paper/2310.02540/) (2023)
- [IIFE: Interaction Information Based Automated Feature Engineering](https://mlsanity.com/paper/2409.04665/) (2024)
- [Regression Oracles and Exploration Strategies for Short-Horizon Multi-Armed Bandits](https://mlsanity.com/paper/2102.05263/) (2021)

## Built on by (most cited first, 3 of 3)

- [Human-LLM Collaborative Feature Engineering for Tabular Data](https://mlsanity.com/paper/2601.21060/) (2026)
- [Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences](https://mlsanity.com/paper/2603.15713/) (2026)
- [Long Text to Predictive Features: LLM-Guided Blockwise Feature Engineering via Executable Program Search](https://mlsanity.com/paper/2610.12390/) (2026)

## Abstract (arXiv)

Feature engineering for tabular data remains a critical yet challenging step in machine learning. Recently, large language models (LLMs) have been used to automatically generate new features by leveraging their vast knowledge. However, existing LLM-based approaches often produce overly simple or repetitive features, partly due to inherent biases in the transformations the LLM chooses and the lack of structured reasoning guidance during generation. In this paper, we propose a novel method REFeat, which guides an LLM to discover diverse and informative features by leveraging multiple types of reasoning to steer the feature generation process. Experiments on 59 benchmark datasets demonstrate that our approach not only achieves higher predictive accuracy on average, but also discovers more diverse and meaningful features. These results highlight the promise of incorporating rich reasoning paradigms and adaptive strategy selection into LLM-driven feature discovery for tabular data.