# Exploring Large Language Models for Feature Selection: A Data-centric Perspective

arXiv: https://arxiv.org/abs/2408.12025 · HTML: https://mlsanity.com/paper/2408.12025/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 75 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 30 of 43)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [ART: Automatic multi-step reasoning and tool-use for large language models](https://mlsanity.com/paper/2303.09014/) (2023)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [Graph Foundation Models: Concepts, Opportunities and Challenges](https://mlsanity.com/paper/2310.11829/) (2023)
- [Revisiting Relation Extraction in the era of Large Language Models](https://mlsanity.com/paper/2305.05003/) (2023)
- [LassoNet: A Neural Network with Feature Sparsity](https://mlsanity.com/paper/1907.12207/) (2019)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [OpenGraph: Towards Open Graph Foundation Models](https://mlsanity.com/paper/2403.01121/) (2024)
- [Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoning](https://mlsanity.com/paper/2403.20046/) (2024)
- [Effective Ways to Build and Evaluate Individual Survival Distributions](https://mlsanity.com/paper/1811.11347/) (2018)
- [MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs](https://mlsanity.com/paper/2406.08772/) (2024)

## Built on by (most cited first, 27 of 27)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Preference Leakage: A Contamination Problem in LLM-as-a-judge](https://mlsanity.com/paper/2502.01534/) (2025)
- [AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML](https://mlsanity.com/paper/2410.02958/) (2024)
- [SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning](https://mlsanity.com/paper/2410.17238/) (2024)
- [Large Language Model-based Data Science Agent: A Survey](https://mlsanity.com/paper/2508.02744/) (2025)
- [LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization](https://mlsanity.com/paper/2502.10648/) (2025)
- [Knowledge-Driven Feature Selection and Engineering for Genotype Data with Large Language Models](https://mlsanity.com/paper/2410.01795/) (2024)
- [AUTOCT: Automating Interpretable Clinical Trial Prediction with LLM Agents](https://mlsanity.com/paper/2506.04293/) (2025)
- [LLMs as World Models: Data-Driven and Human-Centered Pre-Event Simulation for Disaster Impact Assessment](https://mlsanity.com/paper/2506.06355/) (2025)
- [Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judges](https://mlsanity.com/paper/2508.00454/) (2025)
- [LLM4FS: Leveraging Large Language Models for Feature Selection](https://mlsanity.com/paper/2503.24157/) (2025)
- [Comprehend, Divide, and Conquer: Feature Subspace Exploration via Multi-Agent Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2504.17356/) (2025)
- [Multi-level Diagnosis and Evaluation for Robust Tabular Feature Engineering with Large Language Models](https://mlsanity.com/paper/2509.25207/) (2025)
- [LLM-Driven Reasoning for Constraint-Aware Feature Selection in Industrial Systems](https://mlsanity.com/paper/2603.24979/) (2026)
- [LAMP: Extracting Local Decision Surfaces From Large Language Models](https://mlsanity.com/paper/2505.11772/) (2025)
- [PartnerMAS: An LLM Hierarchical Multi-Agent Framework for Business Partner Selection on High-Dimensional Features](https://mlsanity.com/paper/2509.24046/) (2025)
- [LLM-FS-Agent: A Deliberative Role-based Large Language Model Architecture for Transparent Feature Selection](https://mlsanity.com/paper/2510.05935/) (2025)
- [DeepFeature: LLM-Empowered Context-aware Feature Generation for Wearable Biosignals](https://mlsanity.com/paper/2512.08379/) (2025)
- [ReToP: Learning to Rewrite Electronic Health Records for Clinical Prediction](https://mlsanity.com/paper/2601.19286/) (2026)
- [Learning When to Trust LLM Priors: A Validated Framework for Semantic Prior Integration](https://mlsanity.com/paper/2601.21410/) (2026)
- [Hippasus: Effective and Efficient Automatic Feature Augmentation for Machine Learning Tasks on Relational Data](https://mlsanity.com/paper/2602.02025/) (2026)
- [Detecting Jailbreak Attempts in Clinical Training LLMs Through Automated Linguistic Feature Extraction](https://mlsanity.com/paper/2602.13321/) (2026)
- [Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge](https://mlsanity.com/paper/2603.04417/) (2026)
- [BoolXLLM: LLM-Assisted Explainability for Boolean Models](https://mlsanity.com/paper/2605.12139/) (2026)
- [Feature Generation Using LLMs: An Evolutionary Algorithm Approach](https://mlsanity.com/paper/2607.16255/) (2026)
- [When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction](https://mlsanity.com/paper/2608.30086/) (2026)
- [LLM-Enhanced Multi-Agent Reinforcement Learning for Unified Electric Vehicles-Charging Station-Grid Optimization in Public Charging Systems](https://mlsanity.com/paper/2609.13805/) (2026)

## Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) has significantly influenced various domains, leveraging their exceptional few-shot and zero-shot learning capabilities. In this work, we aim to explore and understand the LLMs-based feature selection methods from a data-centric perspective. We begin by categorizing existing feature selection methods with LLMs into two groups: data-driven feature selection which requires numerical values of samples to do statistical inference and text-based feature selection which utilizes prior knowledge of LLMs to do semantical associations using descriptive context. We conduct experiments in both classification and regression tasks with LLMs in various sizes (e.g., GPT-4, ChatGPT and LLaMA-2). Our findings emphasize the effectiveness and robustness of text-based feature selection methods and showcase their potentials using a real-world medical application. We also discuss the challenges and future opportunities in employing LLMs for feature selection, offering insights for further research and development in this emerging field.