mlsanityData to 14 Oct 2026

Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey

arXiv 2411.10478 ↗First posted 11 Nov 2024cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Building effective machine learning (ML) workflows to address complex tasks is a primary focus of the Automatic ML (AutoML) community and a critical step toward achieving artificial general intelligence (AGI). Recently, the integration of Large Language Models (LLMs) into ML workflows has shown great potential for automating and enhancing various stages of the ML pipeline. This survey provides a comprehensive and up-to-date review of recent advancements in using LLMs to construct and optimize ML workflows, focusing on key components encompassing data and feature engineering, model selection and hyperparameter optimization, and workflow evaluation. We discuss both the advantages and limitations of LLM-driven approaches, emphasizing their capacity to streamline and enhance ML workflow modeling process through language understanding, reasoning, interaction, and generation. Finally, we highlight open challenges and propose future research directions to advance the effective application of LLMs in ML workflows.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 93 of its 134 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

93 of 134 references.

References · 93

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
38 more
  1. 2023PaLM 2 Technical Reportcited by 1,162
  2. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  4. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  5. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  6. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  7. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  8. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  9. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  10. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  11. 2024Large Language Models: A Surveycited by 525
  12. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 500
  13. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  14. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  15. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  16. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  17. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  18. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  19. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  20. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  21. 2018Metrics for Explainable AI: Challenges and Prospectscited by 260
  22. 2020Neural Architecture Search without Trainingcited by 244
  23. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  24. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  25. 2020Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learningcited by 174
  26. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  27. 2024Nemotron-4 340B Technical Reportcited by 144
  28. 2024Large Language Models to Enhance Bayesian Optimizationcited by 135
  29. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  30. 2023Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineeringcited by 127
  31. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  32. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 125
  33. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  34. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 88
  35. 2024AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMLcited by 87
  36. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 81
  37. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  38. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 76

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 11.

19 papers.

Sort:

By area:

  1. 2025AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineagecited by 32
  2. 2025AI Scientists Fail Without Strong Implementation Capabilitycited by 21
  3. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
  4. 2025Reinforcement Learning for Machine Learning Engineering Agentscited by 14
  5. 2026Can We Predict Before Executing Machine Learning Agents?cited by 10
  6. 2025SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Sciencecited by 7
  7. 2025Empowering AI to Generate Better AI Code: Guided Generation of Deep Learning Projects with LLMscited by 2
  8. 2025A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standardscited by 1
  9. 2026ADEMA: A Knowledge-State Orchestration Architecture for Long-Horizon Knowledge Synthesis with LLMAgentscited by 1
  10. 2026OmniPlan: An Adaptive Framework for Timely and Near-Optimal Network Planning Optimizationcited by 1
  11. 2025Optimization Problem Solving Can Transition to Evolutionary Agentic Workflowscited by 0
  12. 2026CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimizationcited by 0
7 more
  1. 2026AutoSurrogate: An LLM-Driven Multi-Agent Framework for Autonomous Construction of Deep Learning Surrogate Models in Subsurface Flowcited by 0
  2. 2026Lazy or Efficient? Towards Accessible Eye-Tracking Event Detection Using LLMscited by 0
  3. 2026Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Datacited by 0
  4. 2026Automatic Causal Fairness Analysis with LLM-Generated Reportingcited by 0
  5. 2026GraphBit: A Graph-based Agentic Framework for Non-Linear Agent Orchestrationcited by 0
  6. 2026Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systemscited by 0
  7. 2026Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluationcited by 0