# PAGED: A Benchmark for Procedural Graphs Extraction from Documents

arXiv: https://arxiv.org/abs/2408.03630 · HTML: https://mlsanity.com/paper/2408.03630/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 56 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning](https://mlsanity.com/paper/2107.02794/) (2021)
- [Reasoning about Goals, Steps, and Temporal Ordering with WikiHow](https://mlsanity.com/paper/2009.07690/) (2020)
- [Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference](https://mlsanity.com/paper/2011.10819/) (2020)
- [A Dataset for Tracking Entities in Open Domain Procedural Text](https://mlsanity.com/paper/2011.08092/) (2020)
- [The TechQA Dataset](https://mlsanity.com/paper/1911.02984/) (2019)
- [Goal-Oriented Script Construction](https://mlsanity.com/paper/2107.13189/) (2021)
- [Few-Shot Table-to-Text Generation with Prototype Memory](https://mlsanity.com/paper/2108.12516/) (2021)
- [Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework](https://mlsanity.com/paper/2109.05897/) (2021)
- [Variational Template Machine for Data-to-Text Generation](https://mlsanity.com/paper/2002.01127/) (2020)
- [Neural Pipeline for Zero-Shot Data-to-Text Generation](https://mlsanity.com/paper/2203.16279/) (2022)
- [An Approach for Process Model Extraction By Multi-Grained Text Classification](https://mlsanity.com/paper/1906.02127/) (2019)
- [PET: An Annotated Dataset for Process Extraction from Natural Language Text](https://mlsanity.com/paper/2203.04860/) (2022)
- [Partially-Aligned Data-to-Text Generation with Distant Supervision](https://mlsanity.com/paper/2010.01268/) (2020)
- [Constructing Flow Graphs from Procedural Cybersecurity Texts](https://mlsanity.com/paper/2105.14357/) (2021)
- [Beyond Rule-based Named Entity Recognition and Relation Extraction for Process Model Generation from Natural Language Text](https://mlsanity.com/paper/2305.03960/) (2023)
- [Leveraging pre-trained language models for conversational information seeking from text](https://mlsanity.com/paper/2204.03542/) (2022)
- [Learning from Multiple Sources for Data-to-Text and Text-to-Data](https://mlsanity.com/paper/2302.11269/) (2023)

## Built on by (most cited first, 4 of 4)

- [Accurate and Consistent Graph Model Generation from Text with Large Language Models](https://mlsanity.com/paper/2508.00255/) (2025)
- [Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs](https://mlsanity.com/paper/2505.18542/) (2025)
- [A Highly Clean Recipe Dataset with Ingredient States Annotation for State Probing Task](https://mlsanity.com/paper/2507.17232/) (2025)
- [Multi-Agent Procedural Graph Extraction with Structural and Logical Refinement](https://mlsanity.com/paper/2601.19170/) (2026)

## Abstract (arXiv)

Automatic extraction of procedural graphs from documents creates a low-cost way for users to easily understand a complex procedure by skimming visual graphs. Despite the progress in recent studies, it remains unanswered: whether the existing studies have well solved this task (Q1) and whether the emerging large language models (LLMs) can bring new opportunities to this task (Q2). To this end, we propose a new benchmark PAGED, equipped with a large high-quality dataset and standard evaluations. It investigates five state-of-the-art baselines, revealing that they fail to extract optimal procedural graphs well because of their heavy reliance on hand-written rules and limited available data. We further involve three advanced LLMs in PAGED and enhance them with a novel self-refine strategy. The results point out the advantages of LLMs in identifying textual elements and their gaps in building logical structures. We hope PAGED can serve as a major landmark for automatic procedural graph extraction and the investigations in PAGED can offer insights into the research on logic reasoning among non-sequential elements.