# LLM4FS: Leveraging Large Language Models for Feature Selection

arXiv: https://arxiv.org/abs/2503.24157 · HTML: https://mlsanity.com/paper/2503.24157/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [LassoNet: A Neural Network with Feature Sparsity](https://mlsanity.com/paper/1907.12207/) (2019)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [LMPriors: Pre-Trained Language Models as Task-Specific Priors](https://mlsanity.com/paper/2210.12530/) (2022)
- [LLM-Select: Feature Selection with Large Language Models](https://mlsanity.com/paper/2407.02694/) (2024)
- [Exploring Large Language Models for Feature Selection: A Data-centric Perspective](https://mlsanity.com/paper/2408.12025/) (2024)
- [LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization](https://mlsanity.com/paper/2502.10648/) (2025)
- [Knowledge-Driven Feature Selection and Engineering for Genotype Data with Large Language Models](https://mlsanity.com/paper/2410.01795/) (2024)
- [ICE-SEARCH: A Language Model-Driven Feature Selection Approach](https://mlsanity.com/paper/2402.18609/) (2024)

## Built on by (most cited first, 2 of 2)

- [LLM-Driven Reasoning for Constraint-Aware Feature Selection in Industrial Systems](https://mlsanity.com/paper/2603.24979/) (2026)
- [Hippasus: Effective and Efficient Automatic Feature Augmentation for Machine Learning Tasks on Relational Data](https://mlsanity.com/paper/2602.02025/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have provided new opportunities for decision-making, particularly in the task of automated feature selection. In this paper, we first comprehensively evaluate LLM-based feature selection methods, covering the state-of-the-art DeepSeek-R1, GPT-o3-mini, and GPT-4.5. Then, we propose a new hybrid strategy called LLM4FS that integrates LLMs with traditional data-driven methods. Specifically, input data samples into LLMs, and directly call traditional data-driven techniques such as random forest and forward sequential selection. Notably, our analysis reveals that the hybrid strategy leverages the contextual understanding of LLMs and the high statistical reliability of traditional data-driven methods to achieve excellent feature selection performance, even surpassing LLMs and traditional data-driven methods. Finally, we point out the limitations of its application in decision-making. Our code is available at https://github.com/xianchaoxiu/LLM4FS.