mlsanityData to 14 Oct 2026

FeRG-LLM : Feature Engineering by Reason Generation Large Language Models

arXiv 2503.23371 ↗First posted 30 Mar 2025cs.CL cs.AIcited by 8

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

One of the key tasks in machine learning for tabular data is feature engineering. Although it is vital for improving the performance of models, it demands considerable human expertise and deep domain knowledge, making it labor-intensive endeavor. To address this issue, we propose a novel framework, FeRG-LLM (Feature engineering by Reason Generation Large Language Models), a large language model designed to automatically perform feature engineering at an 8-billion-parameter scale. We have constructed two-stage conversational dialogues that enable language models to analyze machine learning tasks and discovering new features, exhibiting their Chain-of-Thought (CoT) capabilities. We use these dialogues to fine-tune Llama 3.1 8B model and integrate Direct Preference Optimization (DPO) to receive feedback improving quality of new features and the model's performance. Our experiments show that FeRG-LLM performs comparably to or better than Llama 3.1 70B on most datasets, while using fewer resources and achieving reduced inference time. It outperforms other studies in classification tasks and performs well in regression tasks. Moreover, since it does not rely on cloud-hosted LLMs like GPT-4 with extra API costs when generating features, it can be deployed locally, addressing security concerns.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 7 of its 26 references that are in the map. First posted on arXiv 30 Mar 2025.

Builds on

7 of 26 references.

References · 7

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022Teaching Small Language Models to Reasoncited by 284
  7. 2024Token-level Direct Preference Optimizationcited by 135

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026Human-LLM Collaborative Feature Engineering for Tabular Datacited by 1
  2. 2026FeatEHR-LLM: Leveraging Large Language Models for Feature Engineering in Electronic Health Recordscited by 1
  3. 2026BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classificationcited by 1
  4. 2025PaceLLM: Brain-Inspired Large Language Models for Long-Context Understandingcited by 0
  5. 2025Programmatic Representation Learning with Language Modelscited by 0
  6. 2026CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimizationcited by 0
  7. 2026Bridging Expert Knowledge and Automated Feature Engineering via Self-Evolutioncited by 0
  8. 2026Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers
  2. 2026Feature Generation Using LLMs: An Evolutionary Algorithm Approach
  3. 2025Tabular Feature Discovery With Reasoning Type Exploration
  4. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning
  5. 2026CoFEE: Reasoning Control for LLM-Based Feature Discovery
  6. 2025LLM4FS: Leveraging Large Language Models for Feature Selection
  7. 2025Applications of Large Language Model Reasoning in Feature Generation
  8. 2025Large Language Models as Universal Predictors? An Empirical Study on Small Tabular Datasets