# MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks

arXiv: https://arxiv.org/abs/2304.14979 · HTML: https://mlsanity.com/paper/2304.14979/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 97 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [FLAML: A Fast and Lightweight AutoML Library](https://mlsanity.com/paper/1911.04706/) (2019)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [Towards Learning Universal Hyperparameter Optimizers with Transformers](https://mlsanity.com/paper/2205.13320/) (2022)
- [HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML](https://mlsanity.com/paper/2106.06257/) (2021)
- [Can GPT-4 Perform Neural Architecture Search?](https://mlsanity.com/paper/2304.10970/) (2023)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population](https://mlsanity.com/paper/2201.03335/) (2022)
- [HyperSTAR: Task-Aware Hyperparameters for Deep Networks](https://mlsanity.com/paper/2005.10524/) (2020)
- [LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-training](https://mlsanity.com/paper/2212.02691/) (2022)
- [Privacy-preserving Online AutoML for Domain-Specific Face Detection](https://mlsanity.com/paper/2203.08399/) (2022)

## Built on by (most cited first, 30 of 66)

- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024)
- [C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness](https://mlsanity.com/paper/2412.11664/) (2024)
- [AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML](https://mlsanity.com/paper/2410.02958/) (2024)
- [AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions](https://mlsanity.com/paper/2410.20424/) (2024)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [Large Language Model Agent for Hyper-Parameter Optimization](https://mlsanity.com/paper/2402.01881/) (2024)
- [ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoning](https://mlsanity.com/paper/2506.16499/) (2025)
- [Large Language Models Empowered Autonomous Edge AI for Connected Intelligence](https://mlsanity.com/paper/2307.02779/) (2023)
- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [EXP-Bench: Can AI Conduct AI Research Experiments?](https://mlsanity.com/paper/2505.24785/) (2025)
- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents](https://mlsanity.com/paper/2502.16069/) (2025)
- [InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation](https://mlsanity.com/paper/2407.06423/) (2024)
- [Benchmarking Data Science Agents](https://mlsanity.com/paper/2402.17168/) (2024)
- [LAMBDA: A Large Model Based Data Agent](https://mlsanity.com/paper/2407.17535/) (2024)
- [ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering](https://mlsanity.com/paper/2505.23723/) (2025)
- [Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback](https://mlsanity.com/paper/2501.03916/) (2025)
- [SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning](https://mlsanity.com/paper/2410.17238/) (2024)
- [nvAgent: Automated Data Visualization from Natural Language via Collaborative Agent Workflow](https://mlsanity.com/paper/2502.05036/) (2025)
- [A Survey on Large Language Model-based Agents for Statistics and Data Science](https://mlsanity.com/paper/2412.14222/) (2024)
- [BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Tasks](https://mlsanity.com/paper/2411.07464/) (2024)
- [Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey](https://mlsanity.com/paper/2411.10478/) (2024)
- [AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks](https://mlsanity.com/paper/2508.00890/) (2025)
- [AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modelling](https://mlsanity.com/paper/2411.17284/) (2024)
- [BioKGBench: A Knowledge Graph Checking Benchmark of AI Agent for Biomedical Science](https://mlsanity.com/paper/2407.00466/) (2024)

## Abstract (arXiv)

The field of machine learning (ML) has gained widespread adoption, leading to significant demand for adapting ML to specific scenarios, which is yet expensive and non-trivial. The predominant approaches towards the automation of solving ML tasks (e.g., AutoML) are often time-consuming and hard to understand for human developers. In contrast, though human engineers have the incredible ability to understand tasks and reason about solutions, their experience and knowledge are often sparse and difficult to utilize by quantitative approaches. In this paper, we aim to bridge the gap between machine intelligence and human knowledge by introducing a novel framework, which leverages the state-of-the-art large language models to develop ML solutions for novel tasks. We showcase the possibility of extending the capability of LLMs to comprehend structured inputs and perform thorough reasoning for solving novel ML tasks. And we find that, after some dedicated design, the LLM can (i) observe from the existing experiences of ML tasks and (ii) reason effectively to deliver promising results for new tasks. The solution generated can be used directly to achieve high levels of competitiveness. Examples and code available at https://github.com/microsoft/CoML.