mlsanityData to 14 Oct 2026

SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures

arXiv 2510.08942 ↗First posted 10 Oct 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) are widely deployed as domain-specific agents, many benchmarks have been proposed to evaluate their ability to follow instructions and make decisions in real-world scenarios. However, business scenarios often involve complex standard operating procedures (SOPs), and the evaluation of LLM capabilities in such contexts has not been fully explored. To bridge this gap, we propose SOP-Maze, a benchmark constructed from real-world business data and adapted into a collection of 397 instances and 3422 subtasks from 23 complex SOP scenarios. We further categorize SOP tasks into two broad classes: Lateral Root System (LRS), representing wide-option tasks that demand precise selection; and Heart Root System (HRS), which emphasizes deep logical reasoning with complex branches. Extensive experiments reveal that nearly all state-of-the-art models struggle with SOP-Maze. We conduct a comprehensive analysis and identify three key error categories: (i) route blindness: difficulty following procedures; (ii) conversational fragility: inability to handle real dialogue nuances; and (iii) calculation errors: mistakes in time or arithmetic reasoning under complex contexts. The systematic study explores LLM performance across SOP tasks that challenge both breadth and depth, offering new insights for improving model capabilities. We have open-sourced our work on: https://github.com/meituan-longcat/SOP-Maze.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

21 of 31 references.

References · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  5. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  6. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 152
  7. 2024Large Language Models Meet NLP: A Surveycited by 103
  8. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 80
  9. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 70
  10. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  11. 2023Large Language Models can accomplish Business Process Management Taskscited by 34
  12. 2024History, Development, and Principles of Large Language Models-An Introductory Surveycited by 30
9 more
  1. 2025StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Followingcited by 29
  2. 2025SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraintscited by 23
  3. 2025SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agentscited by 23
  4. 2024PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Taskscited by 16
  5. 2024Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Taskscited by 14
  6. 2025GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agentscited by 14
  7. 2024Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysiscited by 13
  8. 2024Towards a Benchmark for Causal Business Process Reasoning with LLMscited by 7
  9. 2024Leveraging Large Language Models for Enhanced Process Model Comprehensioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimizationcited by 1
  2. 2026Eluna: An Agentic LLM System for Automating Warehouse Operations with Reasoning and Task Executioncited by 1
  3. 2026RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judgescited by 1
  4. 2026ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FM SO.P: A Progressive Task Mixture Framework with Automatic Evaluation for Cross-Domain SOP Understanding
  2. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  3. 2025Enterprise Large Language Model Evaluation Benchmark
  4. 2024LLM2: Let Large Language Models Harness System 2 Reasoning
  5. 2025Reasoning LLMs are Wandering Solution Explorers
  6. 2024Easy Problems That LLMs Get Wrong
  7. 2025The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Execution
  8. 2026ReEfBench: Quantifying the Reasoning Efficiency of LLMs