# Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey

arXiv: https://arxiv.org/abs/2411.10478 · HTML: https://mlsanity.com/paper/2411.10478/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 93 of its 134 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Builds on (most cited first, 30 of 93)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)

## Built on by (most cited first, 19 of 19)

- [AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage](https://mlsanity.com/paper/2505.20662/) (2025)
- [AI Scientists Fail Without Strong Implementation Capability](https://mlsanity.com/paper/2506.01372/) (2025)
- [AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks](https://mlsanity.com/paper/2508.00890/) (2025)
- [Reinforcement Learning for Machine Learning Engineering Agents](https://mlsanity.com/paper/2509.01684/) (2025)
- [Can We Predict Before Executing Machine Learning Agents?](https://mlsanity.com/paper/2601.05930/) (2026)
- [SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science](https://mlsanity.com/paper/2503.23314/) (2025)
- [Empowering AI to Generate Better AI Code: Guided Generation of Deep Learning Projects with LLMs](https://mlsanity.com/paper/2504.15080/) (2025)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [ADEMA: A Knowledge-State Orchestration Architecture for Long-Horizon Knowledge Synthesis with LLMAgents](https://mlsanity.com/paper/2604.25849/) (2026)
- [OmniPlan: An Adaptive Framework for Timely and Near-Optimal Network Planning Optimization](https://mlsanity.com/paper/2606.18105/) (2026)
- [Optimization Problem Solving Can Transition to Evolutionary Agentic Workflows](https://mlsanity.com/paper/2505.04354/) (2025)
- [CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization](https://mlsanity.com/paper/2602.09851/) (2026)
- [AutoSurrogate: An LLM-Driven Multi-Agent Framework for Autonomous Construction of Deep Learning Surrogate Models in Subsurface Flow](https://mlsanity.com/paper/2604.11945/) (2026)
- [Lazy or Efficient? Towards Accessible Eye-Tracking Event Detection Using LLMs](https://mlsanity.com/paper/2604.13243/) (2026)
- [Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data](https://mlsanity.com/paper/2604.20261/) (2026)
- [Automatic Causal Fairness Analysis with LLM-Generated Reporting](https://mlsanity.com/paper/2604.27011/) (2026)
- [GraphBit: A Graph-based Agentic Framework for Non-Linear Agent Orchestration](https://mlsanity.com/paper/2605.13848/) (2026)
- [Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems](https://mlsanity.com/paper/2606.25533/) (2026)
- [Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluation](https://mlsanity.com/paper/2609.35335/) (2026)

## Abstract (arXiv)

Building effective machine learning (ML) workflows to address complex tasks is a primary focus of the Automatic ML (AutoML) community and a critical step toward achieving artificial general intelligence (AGI). Recently, the integration of Large Language Models (LLMs) into ML workflows has shown great potential for automating and enhancing various stages of the ML pipeline. This survey provides a comprehensive and up-to-date review of recent advancements in using LLMs to construct and optimize ML workflows, focusing on key components encompassing data and feature engineering, model selection and hyperparameter optimization, and workflow evaluation. We discuss both the advantages and limitations of LLM-driven approaches, emphasizing their capacity to streamline and enhance ML workflow modeling process through language understanding, reasoning, interaction, and generation. Finally, we highlight open challenges and propose future research directions to advance the effective application of LLMs in ML workflows.