# BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation

arXiv: https://arxiv.org/abs/2109.06850 · HTML: https://mlsanity.com/paper/2109.06850/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases](https://mlsanity.com/paper/2009.11564/) (2020)
- [ExplainaBoard: An Explainable Leaderboard for NLP](https://mlsanity.com/paper/2104.06387/) (2021)
- [Span Model for Open Information Extraction on Accurate Corpus](https://mlsanity.com/paper/1901.10879/) (2019)
- [Multi2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT](https://mlsanity.com/paper/2009.08128/) (2020)
- [OPIEC: An Open Information Extraction Corpus](https://mlsanity.com/paper/1904.12324/) (2019)
- [Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction](https://mlsanity.com/paper/2004.08134/) (2020)
- [WiRe57 : A Fine-Grained Benchmark for Open Information Extraction](https://mlsanity.com/paper/1809.08962/) (2018)
- [DocOIE: A Document-level Context-Aware Dataset for OpenIE](https://mlsanity.com/paper/2105.04271/) (2021)
- [Analysing Errors of Open Information Extraction Systems](https://mlsanity.com/paper/1707.07499/) (2017)
- [Assertion-based QA with Question-Aware Open Information Extraction](https://mlsanity.com/paper/1801.07414/) (2018)

## Built on by (most cited first, 11 of 11)

- [Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction](https://mlsanity.com/paper/2305.13981/) (2023)
- [AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark](https://mlsanity.com/paper/2109.07464/) (2021)
- [Linking Surface Facts to Large-Scale Knowledge Graphs](https://mlsanity.com/paper/2310.14909/) (2023)
- [KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models](https://mlsanity.com/paper/2208.11024/) (2022)
- [Leveraging Open Information Extraction for More Robust Domain Transfer of Event Trigger Detection](https://mlsanity.com/paper/2305.14163/) (2023)
- [Open Information Extraction via Chunks](https://mlsanity.com/paper/2305.03299/) (2023)
- [Rules still work for Open Information Extraction](https://mlsanity.com/paper/2403.10758/) (2024)
- [textitBenchIEFL : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark](https://mlsanity.com/paper/2407.16860/) (2024)
- [TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action](https://mlsanity.com/paper/2509.15098/) (2025)
- [KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text](https://mlsanity.com/paper/2607.10212/) (2026)
- [DiffIE: Diffusion-based Open Information Extraction](https://mlsanity.com/paper/2609.02315/) (2026)

## Abstract (arXiv)

Intrinsic evaluations of OIE systems are carried out either manually -- with human evaluators judging the correctness of extractions -- or automatically, on standardized benchmarks. The latter, while much more cost-effective, is less reliable, primarily because of the incompleteness of the existing OIE benchmarks: the ground truth extractions do not include all acceptable variants of the same fact, leading to unreliable assessment of the models' performance. Moreover, the existing OIE benchmarks are available for English only. In this work, we introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese, and German. In contrast to existing OIE benchmarks, BenchIE is fact-based, i.e., it takes into account informational equivalence of extractions: our gold standard consists of fact synsets, clusters in which we exhaustively list all acceptable surface forms of the same fact. Moreover, having in mind common downstream applications for OIE, we make BenchIE multi-faceted; i.e., we create benchmark variants that focus on different facets of OIE evaluation, e.g., compactness or minimality of extractions. We benchmark several state-of-the-art OIE systems using BenchIE and demonstrate that these systems are significantly less effective than indicated by existing OIE benchmarks. We make BenchIE (data and evaluation code) publicly available on https://github.com/gkiril/benchie.