# SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures

arXiv: https://arxiv.org/abs/2510.08942 · HTML: https://mlsanity.com/paper/2510.08942/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach](https://mlsanity.com/paper/2305.07001/) (2023)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability](https://mlsanity.com/paper/2402.18667/) (2024)
- [COLLIE: Systematic Construction of Constrained Text Generation Tasks](https://mlsanity.com/paper/2307.08689/) (2023)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)
- [Large Language Models can accomplish Business Process Management Tasks](https://mlsanity.com/paper/2307.09923/) (2023)
- [History, Development, and Principles of Large Language Models-An Introductory Survey](https://mlsanity.com/paper/2402.06853/) (2024)
- [StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following](https://mlsanity.com/paper/2502.14494/) (2025)
- [SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints](https://mlsanity.com/paper/2503.08669/) (2025)
- [SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents](https://mlsanity.com/paper/2506.08119/) (2025)
- [PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks](https://mlsanity.com/paper/2407.13244/) (2024)
- [Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Tasks](https://mlsanity.com/paper/2407.02310/) (2024)
- [GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agents](https://mlsanity.com/paper/2505.11368/) (2025)
- [Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis](https://mlsanity.com/paper/2412.00023/) (2024)
- [Towards a Benchmark for Causal Business Process Reasoning with LLMs](https://mlsanity.com/paper/2406.05506/) (2024)
- [Leveraging Large Language Models for Enhanced Process Model Comprehension](https://mlsanity.com/paper/2408.08892/) (2024)

## Built on by (most cited first, 4 of 4)

- [Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization](https://mlsanity.com/paper/2605.13641/) (2026)
- [Eluna: An Agentic LLM System for Automating Warehouse Operations with Reasoning and Task Execution](https://mlsanity.com/paper/2607.08960/) (2026)
- [RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges](https://mlsanity.com/paper/2608.24231/) (2026)
- [ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models](https://mlsanity.com/paper/2610.03356/) (2026)

## Abstract (arXiv)

As large language models (LLMs) are widely deployed as domain-specific agents, many benchmarks have been proposed to evaluate their ability to follow instructions and make decisions in real-world scenarios. However, business scenarios often involve complex standard operating procedures (SOPs), and the evaluation of LLM capabilities in such contexts has not been fully explored. To bridge this gap, we propose SOP-Maze, a benchmark constructed from real-world business data and adapted into a collection of 397 instances and 3422 subtasks from 23 complex SOP scenarios. We further categorize SOP tasks into two broad classes: Lateral Root System (LRS), representing wide-option tasks that demand precise selection; and Heart Root System (HRS), which emphasizes deep logical reasoning with complex branches. Extensive experiments reveal that nearly all state-of-the-art models struggle with SOP-Maze. We conduct a comprehensive analysis and identify three key error categories: (i) route blindness: difficulty following procedures; (ii) conversational fragility: inability to handle real dialogue nuances; and (iii) calculation errors: mistakes in time or arithmetic reasoning under complex contexts. The systematic study explores LLM performance across SOP tasks that challenge both breadth and depth, offering new insights for improving model capabilities. We have open-sourced our work on: https://github.com/meituan-longcat/SOP-Maze.