# WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models

arXiv: https://arxiv.org/abs/2411.05451 · HTML: https://mlsanity.com/paper/2411.05451/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [WebCPM: Interactive Web Search for Chinese Long-form Question Answering](https://mlsanity.com/paper/2305.06849/) (2023)
- [AutoFlow: Automated Workflow Generation for Large Language Model Agents](https://mlsanity.com/paper/2407.12821/) (2024)
- [Is GPT-4 a Good Data Analyst?](https://mlsanity.com/paper/2305.15038/) (2023)
- [OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation](https://mlsanity.com/paper/2407.19056/) (2024)
- [Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://mlsanity.com/paper/2306.11489/) (2023)
- [ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents](https://mlsanity.com/paper/2407.00132/) (2024)
- [ProAgent: From Robotic Process Automation to Agentic Process Automation](https://mlsanity.com/paper/2311.10751/) (2023)
- [Automating the Enterprise with Foundation Models](https://mlsanity.com/paper/2405.03710/) (2024)
- [Robotic Process Automation -- A Systematic Literature Review and Assessment Framework](https://mlsanity.com/paper/2012.11951/) (2020)
- [Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts](https://mlsanity.com/paper/2404.02475/) (2024)
- [On the Evaluation of Intelligent Process Automation](https://mlsanity.com/paper/2001.02639/) (2020)
- [Towards Intelligent Robotic Process Automation for BPMers](https://mlsanity.com/paper/2001.00804/) (2020)

## Built on by (most cited first, 30 of 30)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents](https://mlsanity.com/paper/2603.22386/) (2026)
- [A Survey on Agent Workflow -- Status and Future](https://mlsanity.com/paper/2508.01186/) (2025)
- [HALO: Hierarchical Autonomous Logic-Oriented Orchestration for Multi-Agent LLM Systems](https://mlsanity.com/paper/2505.13516/) (2025)
- [Agentic Knowledgeable Self-awareness](https://mlsanity.com/paper/2504.03553/) (2025)
- [Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost](https://mlsanity.com/paper/2605.22502/) (2026)
- [Watermarking LLM Agent Trajectories](https://mlsanity.com/paper/2602.18700/) (2026)
- [StarFlow: Generating Structured Workflow Outputs From Sketch Images](https://mlsanity.com/paper/2503.21889/) (2025)
- [Guideline Forest: Retrieval-Augmented Reasoning with Branching Experience-Induced Guidelines](https://mlsanity.com/paper/2506.07820/) (2025)
- [GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving](https://mlsanity.com/paper/2605.22566/) (2026)
- [LawFlow: Collecting and Simulating Lawyers' Thought Processes on Business Formation Case Studies](https://mlsanity.com/paper/2504.18942/) (2025)
- [In-Context Prompting Obsoletes Agent Orchestration for Procedural Tasks](https://mlsanity.com/paper/2604.27891/) (2026)
- [From Agent Loops to Deterministic Graphs: Execution Lineage for Reproducible AI-Native Work](https://mlsanity.com/paper/2605.06365/) (2026)
- [SymbOmni: Evolving Agentic Omni Models via Symbolic Concept Learning](https://mlsanity.com/paper/2607.12042/) (2026)
- [Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs](https://mlsanity.com/paper/2506.10989/) (2025)
- [LEMON: Learning Executable Multi-Agent Orchestration via Counterfactual Reinforcement Learning](https://mlsanity.com/paper/2605.14483/) (2026)
- [Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraints](https://mlsanity.com/paper/2605.19140/) (2026)
- [SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks](https://mlsanity.com/paper/2608.01117/) (2026)
- [Context-Aware Scientific Knowledge Extraction on Linked Open Data using Large Language Models](https://mlsanity.com/paper/2506.17580/) (2025)
- [Reinforcement Networks: novel framework for collaborative Multi-Agent Reinforcement Learning tasks](https://mlsanity.com/paper/2512.22876/) (2025)
- [FlowMind: Execute-Summarize for Structured Workflow Generation from LLM Reasoning](https://mlsanity.com/paper/2602.11782/) (2026)
- [Intermediate Artifacts as First-Class Citizens: A Data Model for Durable Intermediate Artifacts in Agentic Systems](https://mlsanity.com/paper/2605.12087/) (2026)
- [Characterizing Large Language Model Agentic Workflows: A Study on N8n Ecosystem](https://mlsanity.com/paper/2606.29116/) (2026)
- [Workflow as Knowledge: Semantic Persistence for LLM-Mediated Workflows](https://mlsanity.com/paper/2607.08740/) (2026)
- [Natural-Language Workflows Are Not Software Yet: Artifact-Driven Compilation for Reliable Agent Execution](https://mlsanity.com/paper/2608.21341/) (2026)
- [Generating Workflow DAGs from Natural Language with Non-Reasoning LLMs](https://mlsanity.com/paper/2608.30250/) (2026)
- [Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures](https://mlsanity.com/paper/2609.34215/) (2026)
- [Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses](https://mlsanity.com/paper/2609.36630/) (2026)
- [MoFlow: Multi-Objective Agentic Workflow Generation](https://mlsanity.com/paper/2609.38294/) (2026)
- [MAS-Gate: Unlocking Multi-Agent System Lifecycle via Node-Centric Quantification](https://mlsanity.com/paper/2610.17613/) (2026)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) have driven a revolutionary paradigm shift in process automation from Robotic Process Automation to Agentic Process Automation by automating the workflow orchestration procedure based on LLMs. However, existing LLMs (even the advanced OpenAI GPT-4o) are confined to achieving satisfactory capability in workflow orchestration. To address this limitation, we present WorkflowLLM, a data-centric framework elaborately designed to enhance the capability of LLMs in workflow orchestration. It first constructs a large-scale fine-tuning dataset WorkflowBench with 106,763 samples, covering 1,503 APIs from 83 applications across 28 categories. Specifically, the construction process can be divided into three phases: (1) Data Collection: we collect real-world workflow data from Apple Shortcuts and RoutineHub, transcribing them into Python-style code. We further equip them with generated hierarchical thought via ChatGPT. (2) Query Expansion: we prompt ChatGPT to generate more task queries to enrich the diversity and complexity of workflows. (3) Workflow Generation: we leverage an annotator model trained on collected data to generate workflows for synthesized queries. Finally, we merge the synthetic samples that pass quality confirmation with the collected samples to obtain the WorkflowBench. Based on WorkflowBench, we fine-tune Llama-3.1-8B to obtain WorkflowLlama. Our experiments show that WorkflowLlama demonstrates a strong capacity to orchestrate complex workflows, while also achieving notable generalization performance on previously unseen APIs. Additionally, WorkflowBench exhibits robust zero-shot generalization capabilities on an out-of-distribution task planning dataset, T-Eval. Our data and code are available at https://github.com/OpenBMB/WorkflowLLM.