mlsanityData to 7 Oct 2026

Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning

arXiv 2404.09491 ↗First posted 15 Apr 2024cs.LGcited by 79

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs), with their remarkable ability to tackle challenging and unseen reasoning problems, hold immense potential for tabular learning, that is vital for many real-world applications. In this paper, we propose a novel in-context learning framework, FeatLLM, which employs LLMs as feature engineers to produce an input data set that is optimally suited for tabular predictions. The generated features are used to infer class likelihood with a simple downstream machine learning model, such as linear regression and yields high performance few-shot learning. The proposed FeatLLM framework only uses this simple predictive model with the discovered features at inference time. Compared to existing LLM-based approaches, FeatLLM eliminates the need to send queries to the LLM for each sample at inference time. Moreover, it merely requires API-level access to LLMs, and overcomes prompt size limitations. As demonstrated across numerous tabular datasets from a wide range of domains, FeatLLM generates high-quality rules, significantly (10% on average) outperforming alternatives such as TabLLM and STUNT.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

37 of 52 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
25 more
  1. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2022Galactica: A Large Language Model for Sciencecited by 736
  4. 2019A Closer Look at Few-shot Classificationcited by 679
  5. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  6. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  7. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  8. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  9. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  10. 2021Deep learning: a statistical viewpointcited by 239
  11. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  12. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  13. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  14. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  15. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  16. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 101
  17. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  18. 2022Automated Repair of Programs from Large Language Modelscited by 71
  19. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  20. 2022Transfer Learning with Deep Tabular Modelscited by 52
  21. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  22. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 48
  23. 2023Large Language Models as Superpositions of Cultural Perspectivescited by 48
  24. 2023TABLET: Learning From Instructions For Tabular Datacited by 28
  25. 2023Leveraging Large Language Models for Automated Dialogue Analysiscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 34; 2026: 37.

79 papers.

Sort:

By area:

  1. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  2. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  3. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  4. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizerscited by 35
  5. 2024Exploring Large Language Models for Feature Selection: A Data-centric Perspectivecited by 27
  6. 2025The FM Agentcited by 20
  7. 2025LLM Embeddings for Deep Learning on Tabular Datacited by 11
  8. 2025HyperG: Hypergraph-Enhanced LLMs for Structured Knowledgecited by 11
  9. 2024ELF-Gym: Evaluating Large Language Models Generated Features for Tabular Predictioncited by 10
  10. 2025LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularizationcited by 10
  11. 2024Knowledge-Driven Feature Selection and Engineering for Genotype Data with Large Language Modelscited by 9
  12. 2025AUTOCT: Automating Interpretable Clinical Trial Prediction with LLM Agentscited by 9
38 more
  1. 2024Large Language Models Engineer Too Many Simple Features For Tabular Datacited by 8
  2. 2026No Need to Train Your RDB Foundation Modelcited by 8
  3. 2025LLM Meeting Decision Trees on Tabular Datacited by 7
  4. 2024"Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Modelscited by 6
  5. 2025FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modescited by 6
  6. 2025Autonomous Data Agents: A New Opportunity for Smart Datacited by 6
  7. 2025LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformationcited by 5
  8. 2025Transfer Learning of Tabular Data by Finetuning Large Language Modelscited by 4
  9. 2026FORESTLLM: Large Language Models Make Random Forest Great on Few-shot Tabular Learningcited by 4
  10. 2026FAMOSE: A ReAct Approach to Automated Feature Discoverycited by 4
  11. 2024Retrieval-Augmented Feature Generation for Domain-Specific Classificationcited by 3
  12. 2025Integrating Expert Knowledge into Logical Programs via LLMscited by 3
  13. 2025Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learningcited by 3
  14. 2025Tabular Feature Discovery With Reasoning Type Explorationcited by 3
  15. 2026ReFuGe: Feature Generation for Prediction Tasks on Relational Databases with LLM Agentscited by 3
  16. 2024Cross-Modal Few-Shot Learning: a Generative Transfer Learning Frameworkcited by 2
  17. 2025Architectural and Inferential Inductive Biases For Exchangeable Sequence Modelingcited by 2
  18. 2025LLM4FS: Leveraging Large Language Models for Feature Selectioncited by 2
  19. 2025Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboardcited by 2
  20. 2025LLM Empowered Prototype Learning for Zero and Few-Shot Tasks on Tabular Datacited by 2
  21. 2025FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Datacited by 2
  22. 2025LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domainscited by 2
  23. 2025Knowledge-Informed Automatic Feature Extraction via Collaborative Large Language Model Agentscited by 2
  24. 2026RelAgent: LLM Agents as Data Scientists for Relational Learningcited by 2
  25. 2025Large Language Models as Attribution Regularizers for Efficient Model Trainingcited by 1
  26. 2025Enabling Few-Shot Alzheimer's Disease Diagnosis on Biomarker Data with Tabular LLMscited by 1
  27. 2025Multi-level Diagnosis and Evaluation for Robust Tabular Feature Engineering with Large Language Modelscited by 1
  28. 2025LLM as an Algorithmist: Enhancing Anomaly Detectors via Programmatic Synthesiscited by 1
  29. 2025Dataforge: Agentic Platform for Autonomous Data Engineeringcited by 1
  30. 2026NSR-Boost: A Neuro-Symbolic Residual Boosting Framework for Industrial Legacy Modelscited by 1
  31. 2026Human-LLM Collaborative Feature Engineering for Tabular Datacited by 1
  32. 2026Language Model Representations for Efficient Few-Shot Tabular Classificationcited by 1
  33. 2026Task Expansion and Cross Refinement for Open-World Conditional Modelingcited by 1
  34. 2026LLM-Driven Reasoning for Constraint-Aware Feature Selection in Industrial Systemscited by 1
  35. 2026FeatEHR-LLM: Leveraging Large Language Models for Feature Engineering in Electronic Health Recordscited by 1
  36. 2026BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classificationcited by 1
  37. 2026Eureka: Intelligent Feature Engineering for Enterprise AI Cloud Resource Demand Predictioncited by 1
  38. 2026LLMs as Feature Engineers for Text-and-Tabular Predictioncited by 1

50 most cited of 79 All 79 (JSON)