mlsanityData to 7 Oct 2026

Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering

arXiv 2305.03403 ↗First posted 5 May 2023cs.AI cs.LGcited by 125

Not on the map. Why?

Cited by 39 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems. We present an approach for doing so by harnessing the power of large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features. Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets -- boosting mean ROC AUC performance from 0.798 to 0.822 across all dataset - similar to the improvement achieved by using a random forest instead of logistic regression on our datasets. Furthermore, CAAFE is interpretable by providing a textual explanation for each generated feature. CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our hrefhttps://github.com/automl/CAAFEcode, a simple hrefhttps://colab.research.google.com/drive/1mCA8xOAJZ4MaBalZvyARTMjhl6RZf0ademo and a hrefhttps://pypi.org/project/caafe/python package.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Builds on 18 of its 76 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

18 of 76 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  8. 2014OpenML: networked science in machine learningcited by 523
  9. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  10. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  11. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  12. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
6 more
  1. 2017Feature Engineering for Predictive Modeling using Reinforcement Learningcited by 78
  2. 2019The autofeat Python Library for Automated Feature Engineering and Selectioncited by 73
  3. 2019OpenML-Python: an extensible Python API for OpenMLcited by 62
  4. 2023Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoptioncited by 21
  5. 2021Automating Data Science: Prospects and Challengescited by 17
  6. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 25; 2025: 41; 2026: 57.

125 papers.

Sort:

By area:

  1. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 119
  2. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  3. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  4. 2024AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMLcited by 84
  5. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 62
  6. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 52
  7. 2025DeepAnalyze: Agentic Large Language Models for Autonomous Data Sciencecited by 49
  8. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoningcited by 48
  9. 2025MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineeringcited by 38
  10. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  11. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizerscited by 35
  12. 2024Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Modelscited by 34
38 more
  1. 2025The FM Agentcited by 20
  2. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  3. 2024Reinforcing Language Agents via Policy Optimization with Action Decompositioncited by 16
  4. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
  5. 2024A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Datacited by 15
  6. 2024AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modellingcited by 15
  7. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 15
  8. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcementcited by 12
  9. 2025Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teamingcited by 11
  10. 2025How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?cited by 10
  11. 2024Knowledge-Driven Feature Selection and Engineering for Genotype Data with Large Language Modelscited by 9
  12. 2024ICE-SEARCH: A Language Model-Driven Feature Selection Approachcited by 8
  13. 2024Large Language Models Synergize with Automated Machine Learningcited by 8
  14. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  15. 2024Automating Traffic Model Enhancement with AI Research Agentcited by 8
  16. 2024Large Language Models Engineer Too Many Simple Features For Tabular Datacited by 8
  17. 2025DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automationcited by 8
  18. 2025SemBench: A Benchmark for Semantic Query Processing Enginescited by 8
  19. 2026AgentX: Towards Agent-Driven Self-Iteration of Industrial Recommender Systemscited by 8
  20. 2025IMPROVE: Iterative Model Pipeline Refinement and Optimization Leveraging LLM Expertscited by 7
  21. 2025SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Sciencecited by 7
  22. 2023Vectorizing string entries for data processing on tables: when are larger language models better?cited by 6
  23. 2024Collaborative Evolving Strategy for Automatic Data-Centric Developmentcited by 6
  24. 2024Data Analysis in the Era of Generative AIcited by 6
  25. 2025AutoG: Towards automatic graph construction from tabular datacited by 6
  26. 2025Autonomous Data Agents: A New Opportunity for Smart Datacited by 6
  27. 2026LongDA: Benchmarking LLM Agents for Long-Document Data Analysiscited by 6
  28. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  29. 2025FastFT: Accelerating Reinforced Feature Transformation via Advanced Exploration Strategiescited by 5
  30. 2025LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformationcited by 5
  31. 2025CoMind: Towards Community-Driven Agents for Machine Learning Engineeringcited by 5
  32. 2026Agentic Bayesian Optimization through Surrogate-Augmented Autoresearchcited by 5
  33. 2025Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizescited by 4
  34. 2025AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Sciencecited by 4
  35. 2026Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Modelscited by 4
  36. 2026FAMOSE: A ReAct Approach to Automated Feature Discoverycited by 4
  37. 2024Retrieval-Augmented Feature Generation for Domain-Specific Classificationcited by 3
  38. 2025Tabular Feature Discovery With Reasoning Type Explorationcited by 3

50 most cited of 125 All 125 (JSON)