# WiRe57 : A Fine-Grained Benchmark for Open Information Extraction

arXiv: https://arxiv.org/abs/1809.08962 · HTML: https://mlsanity.com/paper/1809.08962/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2018.

## Built on by (most cited first, 16 of 16)

- [A Survey on Neural Open Information Extraction: Current Status and Future Directions](https://mlsanity.com/paper/2205.11725/) (2022)
- [LSOIE: A Large-Scale Dataset for Supervised Open Information Extraction](https://mlsanity.com/paper/2101.11177/) (2021)
- [BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation](https://mlsanity.com/paper/2109.06850/) (2021)
- [DetIE: Multilingual Open Information Extraction Inspired by Object Detection](https://mlsanity.com/paper/2206.12514/) (2022)
- [DocOIE: A Document-level Context-Aware Dataset for OpenIE](https://mlsanity.com/paper/2105.04271/) (2021)
- [A Survey on Open Information Extraction from Rule-based Model to Large Language Model](https://mlsanity.com/paper/2208.08690/) (2022)
- [Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction](https://mlsanity.com/paper/2305.13981/) (2023)
- [AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark](https://mlsanity.com/paper/2109.07464/) (2021)
- [CompactIE: Compact Facts in Open Information Extraction](https://mlsanity.com/paper/2205.02880/) (2022)
- [When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications](https://mlsanity.com/paper/2211.08228/) (2022)
- [Common Sense Knowledge Learning for Open Vocabulary Neural Reasoning: A First View into Chronic Disease Literature](https://mlsanity.com/paper/2111.13781/) (2021)
- [Rules still work for Open Information Extraction](https://mlsanity.com/paper/2403.10758/) (2024)
- [textitBenchIEFL : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark](https://mlsanity.com/paper/2407.16860/) (2024)
- [Challenges in Expanding Portuguese Resources: A View from Open Information Extraction](https://mlsanity.com/paper/2501.11851/) (2025)
- [A linguistically-motivated evaluation methodology for unraveling model's abilities in reading comprehension tasks](https://mlsanity.com/paper/2501.17569/) (2025)
- [DiffIE: Diffusion-based Open Information Extraction](https://mlsanity.com/paper/2609.02315/) (2026)

## Abstract (arXiv)

We build a reference for the task of Open Information Extraction, on five documents. We tentatively resolve a number of issues that arise, including inference and granularity. We seek to better pinpoint the requirements for the task. We produce our annotation guidelines specifying what is correct to extract and what is not. In turn, we use this reference to score existing Open IE systems. We address the non-trivial problem of evaluating the extractions produced by systems against the reference tuples, and share our evaluation script. Among seven compared extractors, we find the MinIE system to perform best.