# FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents

arXiv: https://arxiv.org/abs/2406.14884 · HTML: https://mlsanity.com/paper/2406.14884/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 30)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [AutoAgents: A Framework for Automatic Agent Generation](https://mlsanity.com/paper/2309.17288/) (2023)
- [HuatuoGPT, towards Taming Language Model to Be a Doctor](https://mlsanity.com/paper/2305.15075/) (2023)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)
- [Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue](https://mlsanity.com/paper/2308.03549/) (2023)
- [T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step](https://mlsanity.com/paper/2312.14033/) (2023)
- [KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graph](https://mlsanity.com/paper/2402.11163/) (2024)
- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents](https://mlsanity.com/paper/2401.00812/) (2024)
- [Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering](https://mlsanity.com/paper/2404.14741/) (2024)
- [STAR: A Schema-Guided Dialog Dataset for Transfer Learning](https://mlsanity.com/paper/2010.11853/) (2020)
- [Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion](https://mlsanity.com/paper/2311.06318/) (2023)
- [ProAgent: From Robotic Process Automation to Agentic Process Automation](https://mlsanity.com/paper/2311.10751/) (2023)
- [Knowledge-enhanced Agents for Interactive Text Games](https://mlsanity.com/paper/2305.05091/) (2023)
- [Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs](https://mlsanity.com/paper/2404.00942/) (2024)

## Built on by (most cited first, 30 of 40)

- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)
- [Benchmarking Agentic Workflow Generation](https://mlsanity.com/paper/2410.07869/) (2024)
- [From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents](https://mlsanity.com/paper/2603.22386/) (2026)
- [Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agents](https://mlsanity.com/paper/2601.22311/) (2026)
- [Goal Alignment in LLM-Based User Simulators for Conversational AI](https://mlsanity.com/paper/2507.20152/) (2025)
- [SEW: Self-Evolving Agentic Workflows for Automated Code Generation](https://mlsanity.com/paper/2505.18646/) (2025)
- [Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey](https://mlsanity.com/paper/2506.11102/) (2025)
- [Position: The Real Barrier to LLM Agent Usability is Agentic ROI](https://mlsanity.com/paper/2505.17767/) (2025)
- [Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agents](https://mlsanity.com/paper/2506.07388/) (2025)
- [A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions](https://mlsanity.com/paper/2504.16939/) (2025)
- [Agentic Knowledgeable Self-awareness](https://mlsanity.com/paper/2504.03553/) (2025)
- [Reflection-Bench: Evaluating Epistemic Agency in Large Language Models](https://mlsanity.com/paper/2410.16270/) (2024)
- [AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints](https://mlsanity.com/paper/2606.05622/) (2026)
- [ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models](https://mlsanity.com/paper/2505.13176/) (2025)
- [How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective](https://mlsanity.com/paper/2508.01300/) (2025)
- [Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis](https://mlsanity.com/paper/2603.15483/) (2026)
- [From An LLM Swarm To A PDDL-Empowered HIVE: Planning Self-Executed Instructions In A Multi-Modal Jungle](https://mlsanity.com/paper/2412.12839/) (2024)
- [Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluation](https://mlsanity.com/paper/2507.02253/) (2025)
- [Execution-State-Aware LLM Reasoning for Automated Proof-of-Vulnerability Generation](https://mlsanity.com/paper/2602.13574/) (2026)
- [The Scaling Laws of Skills in LLM Agent Systems](https://mlsanity.com/paper/2605.16508/) (2026)
- [GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving](https://mlsanity.com/paper/2605.22566/) (2026)
- [AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents](https://mlsanity.com/paper/2505.01592/) (2025)
- [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025)
- [T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains](https://mlsanity.com/paper/2606.11070/) (2026)
- [A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants](https://mlsanity.com/paper/2510.06223/) (2025)

## Abstract (arXiv)

LLM-based agents have emerged as promising tools, which are crafted to fulfill complex tasks by iterative planning and action. However, these agents are susceptible to undesired planning hallucinations when lacking specific knowledge for expertise-intensive tasks. To address this, preliminary attempts are made to enhance planning reliability by incorporating external workflow-related knowledge. Despite the promise, such infused knowledge is mostly disorganized and diverse in formats, lacking rigorous formalization and comprehensive comparisons. Motivated by this, we formalize different formats of workflow knowledge and present FlowBench, the first benchmark for workflow-guided planning. FlowBench covers 51 different scenarios from 6 domains, with knowledge presented in diverse formats. To assess different LLMs on FlowBench, we design a multi-tiered evaluation framework. We evaluate the efficacy of workflow knowledge across multiple formats, and the results indicate that current LLM agents need considerable improvements for satisfactory planning. We hope that our challenging benchmark can pave the way for future agent planning research.