# Towards a Benchmark for Causal Business Process Reasoning with LLMs

arXiv: https://arxiv.org/abs/2406.05506 · HTML: https://mlsanity.com/paper/2406.05506/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 8 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 14 of 14)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Can Large Language Models Reason and Plan?](https://mlsanity.com/paper/2403.04121/) (2024)
- [Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation](https://mlsanity.com/paper/2305.07375/) (2023)
- [TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions](https://mlsanity.com/paper/2005.00242/) (2020)
- [Large Language Models can accomplish Business Process Management Tasks](https://mlsanity.com/paper/2307.09923/) (2023)
- [Can Foundation Models Talk Causality?](https://mlsanity.com/paper/2206.10591/) (2022)
- [AI-Augmented Business Process Management Systems: A Research Manifesto](https://mlsanity.com/paper/2201.12855/) (2022)
- [Large Language Models for Business Process Management: Opportunities and Challenges](https://mlsanity.com/paper/2304.04309/) (2023)
- [Abstractions, Scenarios, and Prompt Definitions for Process Mining with LLMs: A Case Study](https://mlsanity.com/paper/2307.02194/) (2023)
- [Just Tell Me: Prompt Engineering in Business Process Management](https://mlsanity.com/paper/2304.07183/) (2023)
- [PET: An Annotated Dataset for Process Extraction from Natural Language Text](https://mlsanity.com/paper/2203.04860/) (2022)
- [How well can a large language model explain business processes as perceived by users?](https://mlsanity.com/paper/2401.12846/) (2024)
- [The WHY in Business Processes: Discovery of Causal Execution Dependencies](https://mlsanity.com/paper/2310.14975/) (2023)

## Built on by (most cited first, 7 of 7)

- [PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks](https://mlsanity.com/paper/2407.13244/) (2024)
- [SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures](https://mlsanity.com/paper/2510.08942/) (2025)
- [XABPs: Towards eXplainable Autonomous Business Processes](https://mlsanity.com/paper/2507.23269/) (2025)
- [SAGE: A Service Agent Graph-guided Evaluation Benchmark](https://mlsanity.com/paper/2604.09285/) (2026)
- [Assessing the Business Process Modeling Competences of Large Language Models](https://mlsanity.com/paper/2601.21787/) (2026)
- [The Algebra of Meaning: Why Machines Need Montague More Than Moore's Law](https://mlsanity.com/paper/2510.06559/) (2025)
- [On the Hybrid Nature of ABPMS Process Frames and its Implications on Automated Process Discovery](https://mlsanity.com/paper/2604.22455/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used for boosting organizational efficiency and automating tasks. While not originally designed for complex cognitive processes, recent efforts have further extended to employ LLMs in activities such as reasoning, planning, and decision-making. In business processes, such abilities could be invaluable for leveraging on the massive corpora LLMs have been trained on for gaining deep understanding of such processes. In this work, we plant the seeds for the development of a benchmark to assess the ability of LLMs to reason about causal and process perspectives of business operations. We refer to this view as Causally-augmented Business Processes (BP^C). The core of the benchmark comprises a set of BP^C related situations, a set of questions about these situations, and a set of deductive rules employed to systematically resolve the ground truth answers to these questions. Also with the power of LLMs, the seed is then instantiated into a larger-scale set of domain-specific situations and questions. Reasoning on BP^C is of crucial importance for process interventions and process improvement. Our benchmark, accessible at https://huggingface.co/datasets/ibm/BPC, can be used in one of two possible modalities: testing the performance of any target LLM and training an LLM to advance its capability to reason about BP^C.