# LLM-Select: Feature Selection with Large Language Models

arXiv: https://arxiv.org/abs/2407.02694 · HTML: https://mlsanity.com/paper/2407.02694/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2024.

## Built on by (most cited first, 30 of 30)

- [AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage](https://mlsanity.com/paper/2505.20662/) (2025)
- [Exploring Large Language Models for Feature Selection: A Data-centric Perspective](https://mlsanity.com/paper/2408.12025/) (2024)
- [Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey](https://mlsanity.com/paper/2411.10478/) (2024)
- [Large Language Models Engineer Too Many Simple Features For Tabular Data](https://mlsanity.com/paper/2410.17787/) (2024)
- [Knowledge-Driven Feature Selection and Engineering for Genotype Data with Large Language Models](https://mlsanity.com/paper/2410.01795/) (2024)
- [LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation](https://mlsanity.com/paper/2411.00773/) (2024)
- [FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes](https://mlsanity.com/paper/2506.03278/) (2025)
- [LLM4FS: Leveraging Large Language Models for Feature Selection](https://mlsanity.com/paper/2503.24157/) (2025)
- [Collaborative Multi-Agent Reinforcement Learning for Automated Feature Transformation with Graph-Driven Path Optimization](https://mlsanity.com/paper/2504.17355/) (2025)
- [Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth](https://mlsanity.com/paper/2506.06991/) (2025)
- [Robustness is Important: Limitations of LLMs for Data Fitting](https://mlsanity.com/paper/2508.19563/) (2025)
- [Configuration-to-Performance Scaling Law with Neural Ansatz](https://mlsanity.com/paper/2602.10300/) (2026)
- [LLMs can construct powerful representations and streamline sample-efficient supervised learning](https://mlsanity.com/paper/2603.11679/) (2026)
- [LLM-FS: Zero-Shot Feature Selection for Effective and Interpretable Malware Detection](https://mlsanity.com/paper/2602.09634/) (2026)
- [LLM-Driven Reasoning for Constraint-Aware Feature Selection in Industrial Systems](https://mlsanity.com/paper/2603.24979/) (2026)
- [Visual Error Patterns in Multi-Modal AI: A Statistical Approach](https://mlsanity.com/paper/2412.00083/) (2024)
- [LAMP: Extracting Local Decision Surfaces From Large Language Models](https://mlsanity.com/paper/2505.11772/) (2025)
- [InterFeat: A Pipeline for Finding Interesting Scientific Features](https://mlsanity.com/paper/2505.13534/) (2025)
- [PartnerMAS: An LLM Hierarchical Multi-Agent Framework for Business Partner Selection on High-Dimensional Features](https://mlsanity.com/paper/2509.24046/) (2025)
- [LLM-FS-Agent: A Deliberative Role-based Large Language Model Architecture for Transparent Feature Selection](https://mlsanity.com/paper/2510.05935/) (2025)
- [Learning When to Trust LLM Priors: A Validated Framework for Semantic Prior Integration](https://mlsanity.com/paper/2601.21410/) (2026)
- [Mamba-SSM with LLM Reasoning for Feature Selection: Faithfulness-Aware Biomarker Discovery](https://mlsanity.com/paper/2604.14334/) (2026)
- [LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues](https://mlsanity.com/paper/2604.19464/) (2026)
- [LLM Sparsity Prior for Robust Feature Selection](https://mlsanity.com/paper/2605.23102/) (2026)
- [The Daily Dose: Workflow-Integrated Large Language Model Automation for Clinical Summarization and Trial Identification in Radiation Oncology](https://mlsanity.com/paper/2605.26346/) (2026)
- [Feature Generation Using LLMs: An Evolutionary Algorithm Approach](https://mlsanity.com/paper/2607.16255/) (2026)
- [A Comparative Study of Feature Selection Methods for EHR Diagnosis Codes in Opioid Use Disorder Prediction](https://mlsanity.com/paper/2608.04180/) (2026)
- [When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction](https://mlsanity.com/paper/2608.30086/) (2026)
- [LLM-Enhanced Multi-Agent Reinforcement Learning for Unified Electric Vehicles-Charging Station-Grid Optimization in Public Charging Systems](https://mlsanity.com/paper/2609.13805/) (2026)
- [Learning to Select Features with Manifold Deep Unfolding PCA](https://mlsanity.com/paper/2610.19436/) (2026)

## Abstract (arXiv)

In this paper, we demonstrate a surprising capability of large language models (LLMs): given only input feature names and a description of a prediction task, they are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Remarkably, these models exhibit this capacity across various query mechanisms. For example, we zero-shot prompt an LLM to output a numerical importance score for a feature (e.g., "blood pressure") in predicting an outcome of interest (e.g., "heart failure"), with no additional context. In particular, we find that the latest models, such as GPT-4, can consistently identify the most predictive features regardless of the query mechanism and across various prompting strategies. We illustrate these findings through extensive experiments on real-world data, where we show that LLM-based feature selection consistently achieves strong performance competitive with data-driven methods such as the LASSO, despite never having looked at the downstream training data. Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place. This could benefit practitioners in domains like healthcare and the social sciences, where collecting high-quality data comes at a high cost.