# Multi-level Diagnosis and Evaluation for Robust Tabular Feature Engineering with Large Language Models

arXiv: https://arxiv.org/abs/2509.25207 · HTML: https://mlsanity.com/paper/2509.25207/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [Large Scale Transfer Learning for Tabular Data via Language Modeling](https://mlsanity.com/paper/2406.12031/) (2024)
- [LMPriors: Pre-Trained Language Models as Task-Specific Priors](https://mlsanity.com/paper/2210.12530/) (2022)
- [Exploring Large Language Models for Feature Selection: A Data-centric Perspective](https://mlsanity.com/paper/2408.12025/) (2024)
- [MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinement](https://mlsanity.com/paper/2305.12081/) (2023)
- [Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)](https://mlsanity.com/paper/2407.12858/) (2024)
- [Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution](https://mlsanity.com/paper/2408.10548/) (2024)
- [ELF-Gym: Evaluating Large Language Models Generated Features for Tabular Prediction](https://mlsanity.com/paper/2410.12865/) (2024)

## Built on by (most cited first, 1 of 1)

- [Segment-driven Structural Induction and Semantic Alignment for Heterogeneous Tabular Representation](https://mlsanity.com/paper/2606.01890/) (2026)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) have shown promise in feature engineering for tabular data, but concerns about their reliability persist, especially due to variability in generated outputs. We introduce a multi-level diagnosis and evaluation framework to assess the robustness of LLMs in feature engineering across diverse domains, focusing on the three main factors: key variables, relationships, and decision boundary values for predicting target classes. We demonstrate that the robustness of LLMs varies significantly over different datasets, and that high-quality LLM-generated features can improve few-shot prediction performance by up to 10.52%. This work opens a new direction for assessing and enhancing the reliability of LLM-driven feature engineering in various domains.