mlsanityData to 7 Oct 2026

Causal Reasoning of Entities and Events in Procedural Texts

arXiv 2301.10896 ↗First posted 26 Jan 2023cs.CLcited by 23

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

Entities and events are crucial to natural language reasoning and common in procedural texts. Existing work has focused either exclusively on entity state tracking (e.g., whether a pan is hot) or on event reasoning (e.g., whether one would burn themselves by touching the pan), while these two tasks are often causally related. We propose CREPE, the first benchmark on causal reasoning of event plausibility and entity states. We show that most language models, including GPT-3, perform close to chance at .35 F1, lagging far behind human at .87 F1. We boost model performance to .59 F1 by creatively representing events as programming languages while prompting language models pretrained on code. By injecting the causal relations between entities and events as intermediate reasoning steps in our representation, we further boost the performance to .67 F1. Our findings indicate not only the challenge that CREPE brings for language models, but also the efficacy of code-like prompting combined with chain-of-thought prompting for multihop event reasoning.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 58 of its 94 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

58 of 94 references.

References · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
38 more
  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  2. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  3. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  6. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  7. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  8. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  9. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  10. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  11. 2019Entity, Relation, and Event Extraction with Contextualized Span Representationscited by 261
  12. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  13. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  14. 2020Event Extraction by Answering (Almost) Natural Questionscited by 177
  15. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  16. 2019Multi-hop Reading Comprehension through Question Decomposition and Rescoringcited by 119
  17. 2021Counterfactual Zero-Shot and Open-Set Visual Recognitioncited by 95
  18. 2019Hierarchical Graph Network for Multi-hop Question Answeringcited by 83
  19. 2018Jointly Multiple Events Extraction via Attention-based Graph Information Aggregationcited by 73
  20. 2019WIQA: A dataset for "What if..." reasoning over procedural textcited by 71
  21. 2020Reasoning about Goals, Steps, and Temporal Ordering with WikiHowcited by 66
  22. 2019Cognitive Graph for Multi-Hop Reading Comprehension at Scalecited by 65
  23. 2019Dynamically Fused Graph Network for Multi-hop Reasoningcited by 63
  24. 2018Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehensioncited by 62
  25. 2018Question Answering by Reasoning Across Documents with Graph Convolutional Networkscited by 56
  26. 2017Simulating Action Dynamics with Neural Process Networkscited by 53
  27. 2019The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structurescited by 48
  28. 2019Revisiting Joint Modeling of Cross-document Entity and Event Coreference Resolutioncited by 43
  29. 2021Visual Goal-Step Inference using wikiHowcited by 42
  30. 2020A Dataset for Tracking Entities in Open Domain Procedural Textcited by 37
  31. 2016Simpler Context-Dependent Logical Forms via Model Projectionscited by 33
  32. 2019Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Textcited by 31
  33. 2021Goal-Oriented Script Constructioncited by 30
  34. 2020Analogous Process Structure Induction for Sub-event Sequence Predictioncited by 26
  35. 2019BAG: Bi-directional Attention Entity Graph Convolutional Network for Multi-hop Reasoning Question Answeringcited by 19
  36. 2020Intent Detection with WikiHowcited by 19
  37. 2019Tracking Discrete and Continuous Entity State for Process Understandingcited by 15
  38. 2021Empowering Language Understanding with Counterfactual Reasoningcited by 13

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 11; 2024: 8; 2025: 4.

23 papers.

Sort:

By area:

  1. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  2. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  3. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  4. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  5. 2024Causality for Large Language Modelscited by 24
  6. 2024On the Limit of Language Models as Planning Formalizerscited by 17
  7. 2023Prompting with Pseudo-Code Instructionscited by 13
  8. 2024CausalEval: Towards Better Causal Reasoning in Language Modelscited by 13
  9. 2024CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Planscited by 10
  10. 2023OpenPI2.0: An Improved Dataset for Entity Tracking in Textscited by 8
  11. 2024Chain and Causal Attention for Efficient Entity Trackingcited by 8
  12. 2023Exploring the Curious Case of Code Promptscited by 6
11 more
  1. 2025EnigmaToM: Improve LLMs' Theory-of-Mind Reasoning Capabilities with Neural Knowledge Base of Entity Statescited by 6
  2. 2025How Far Are LLMs from Symbolic Planners? An NLP-Based Perspectivecited by 6
  3. 2023Do large language models and humans have similar behaviors in causal inference with script knowledge?cited by 3
  4. 2023Code Models are Zero-shot Precondition Reasonerscited by 3
  5. 2024Cascading Large Language Models for Salient Event Graph Generationcited by 3
  6. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  7. 2023Large Language Models as Sous Chefs: Revising Recipes with GPT-3cited by 2
  8. 2023Choice-75: A Dataset on Decision Branching in Script Learningcited by 2
  9. 2024Formulation Comparison for Timeline Construction using LLMscited by 2
  10. 2023Efficient Pre-training for Localized Instruction Generation of Videoscited by 0
  11. 2025FABLE: A Novel Data-Flow Analysis Benchmark on Procedural Text for Large Language Model Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022EvEntS ReaLM: Event Reasoning of Entity States via Language Models
  2. 2024A Comprehensive Evaluation on Event Reasoning of Large Language Models
  3. 2023CRAB: Assessing the Strength of Causal Relationships Between Real-world Events
  4. 2023The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code
  5. 2025Causal Graph based Event Reasoning using Semantic Relation Experts
  6. 2025ACCESS : A Benchmark for Abstract Causal Event Discovery and Reasoning
  7. 2024Enhancing Event Reasoning in Large Language Models through Instruction Fine-Tuning with Semantic Causal Graphs
  8. 2023Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning