# Automatic Error Analysis for Document-level Information Extraction

arXiv: https://arxiv.org/abs/2209.07442 · HTML: https://mlsanity.com/paper/2209.07442/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 11 of 11)

- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [Document-Level Event Argument Extraction by Conditional Generation](https://mlsanity.com/paper/2104.05919/) (2021)
- [Graph Convolution over Pruned Dependency Trees Improves Relation Extraction](https://mlsanity.com/paper/1809.10185/) (2018)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [A General Framework for Information Extraction using Dynamic Span Graphs](https://mlsanity.com/paper/1904.03296/) (2019)
- [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](https://mlsanity.com/paper/1909.09251/) (2019)
- [Distant Supervision for Relation Extraction beyond the Sentence Boundary](https://mlsanity.com/paper/1609.04873/) (2016)
- [CrossCheck: Rapid, Reproducible, and Interpretable Model Evaluation](https://mlsanity.com/paper/2004.07993/) (2020)

## Built on by (most cited first, 10 of 10)

- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [A Unified View of Evaluation Metrics for Structured Prediction](https://mlsanity.com/paper/2310.13793/) (2023)
- [Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing](https://mlsanity.com/paper/2211.04476/) (2022)
- [MultiMUC: Multilingual Template Filling on MUC-4](https://mlsanity.com/paper/2401.16209/) (2024)
- [On Event Individuation for Document-Level Information Extraction](https://mlsanity.com/paper/2212.09702/) (2022)
- [Event-Keyed Summarization](https://mlsanity.com/paper/2402.06973/) (2024)
- [Understanding the Effect of Knowledge Graph Extraction Error on Downstream Graph Analyses: A Case Study on Affiliation Graphs](https://mlsanity.com/paper/2506.12367/) (2025)
- [Probing Representations for Document-level Event Extraction](https://mlsanity.com/paper/2310.15316/) (2023)
- [Are Triggers Needed for Document-Level Event Extraction?](https://mlsanity.com/paper/2411.08708/) (2024)
- [Causal Intervention-based Prompt Debiasing for Event Argument Extraction](https://mlsanity.com/paper/2210.01561/) (2022)

## Abstract (arXiv)

Document-level information extraction (IE) tasks have recently begun to be revisited in earnest using the end-to-end neural network techniques that have been successful on their sentence-level IE counterparts. Evaluation of the approaches, however, has been limited in a number of dimensions. In particular, the precision/recall/F1 scores typically reported provide few insights on the range of errors the models make. We build on the work of Kummerfeld and Klein (2013) to propose a transformation-based framework for automating error analysis in document-level event and (N-ary) relation extraction. We employ our framework to compare two state-of-the-art document-level template-filling approaches on datasets from three domains; and then, to gauge progress in IE since its inception 30 years ago, vs. four systems from the MUC-4 (1992) evaluation.