mlsanityData to 14 Oct 2026

Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models

arXiv 2412.03589 ↗First posted 27 Nov 2024cs.AI cs.CL cs.HCcited by 4

Not on the map. Why?

Abstract (arXiv)

Procedural Knowledge is the know-how expressed in the form of sequences of steps needed to perform some tasks. Procedures are usually described by means of natural language texts, such as recipes or maintenance manuals, possibly spread across different documents and systems, and their interpretation and subsequent execution is often left to the reader. Representing such procedures in a Knowledge Graph (KG) can be the basis to build digital tools to support those users who need to apply or execute them. In this paper, we leverage Large Language Model (LLM) capabilities and propose a prompt engineering approach to extract steps, actions, objects, equipment and temporal information from a textual procedure, in order to populate a Procedural KG according to a pre-defined ontology. We evaluate the KG extraction results by means of a user study, in order to qualitatively and quantitatively assess the perceived quality and usefulness of the LLM-extracted procedural knowledge. We show that LLMs can produce outputs of acceptable quality and we assess the subjective perception of AI by human evaluators.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 57 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

28 of 57 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  5. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  6. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  7. 2023GPTScore: Evaluate as You Desirecited by 361
  8. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  9. 2021Few-NERD: A Few-Shot Named Entity Recognition Datasetcited by 152
  10. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  11. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 120
  12. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 94
16 more
  1. 2022Evaluating Human-Language Model Interactioncited by 92
  2. 2023LLMs4OL: Large Language Models for Ontology Learningcited by 77
  3. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  4. 2020Reasoning about Goals, Steps, and Temporal Ordering with WikiHowcited by 66
  5. 2023Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Textcited by 48
  6. 2024AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotatorscited by 28
  7. 2023BatchEval: Towards Human-like Text Evaluationcited by 17
  8. 2019Learning Household Task Knowledge from WikiHow Descriptionscited by 15
  9. 2024OntoChat: a Framework for Conversational Ontology Engineering using Language Modelscited by 15
  10. 2023Generative Multimodal Entity Linkingcited by 12
  11. 2023Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?cited by 11
  12. 2023Knowledge Engineering using Large Language Modelscited by 9
  13. 2022Reasoning about Procedures with Natural Language Processing: A Tutorialcited by 8
  14. 2021Process Extraction from Text: Benchmarking the State of the Art and Paving the Way for Future Challengescited by 4
  15. 2023Procedural Text Mining with Large Language Modelscited by 2
  16. 2023Expanding the Vocabulary of BERT for Knowledge Base Constructioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Mutual Understanding between People and Systems via Neurosymbolic AI and Knowledge Graphscited by 1
  2. 2026Procedural Knowledge Extraction from Industrial Troubleshooting Guides Using Vision Language Modelscited by 1
  3. 2025Matching Game Preferences Through Dialogical Large Language Models: A Perspectivecited by 0
  4. 2026Learning from Research: Toward Lifelong Agent Harness Evolutioncited by 0