# Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction

arXiv: https://arxiv.org/abs/2305.13981 · HTML: https://mlsanity.com/paper/2305.13981/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 23 May 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Open Information Extraction](https://mlsanity.com/paper/1805.04270/) (2018)
- [Large-Scale QA-SRL Parsing](https://mlsanity.com/paper/1805.05377/) (2018)
- [ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations](https://mlsanity.com/paper/1711.05732/) (2017)
- [IMoJIE: Iterative Memory-Based Joint Open Information Extraction](https://mlsanity.com/paper/2005.08178/) (2020)
- [Span Model for Open Information Extraction on Accurate Corpus](https://mlsanity.com/paper/1901.10879/) (2019)
- [Multi2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT](https://mlsanity.com/paper/2009.08128/) (2020)
- [Generalizing Natural Language Analysis through Span-relation Representations](https://mlsanity.com/paper/1911.03822/) (2019)
- [RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering](https://mlsanity.com/paper/2210.14353/) (2022)
- [WiRe57 : A Fine-Grained Benchmark for Open Information Extraction](https://mlsanity.com/paper/1809.08962/) (2018)
- [Logician: A Unified End-to-End Neural Approach for Open-Domain Information Extraction](https://mlsanity.com/paper/1904.12535/) (2019)
- [LSOIE: A Large-Scale Dataset for Supervised Open Information Extraction](https://mlsanity.com/paper/2101.11177/) (2021)
- [Knowledge Infused Decoding](https://mlsanity.com/paper/2204.03084/) (2022)
- [OpenIE6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction](https://mlsanity.com/paper/2010.03147/) (2020)
- [BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation](https://mlsanity.com/paper/2109.06850/) (2021)
- [DocOIE: A Document-level Context-Aware Dataset for OpenIE](https://mlsanity.com/paper/2105.04271/) (2021)
- [When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications](https://mlsanity.com/paper/2211.08228/) (2022)
- [Syntactically Robust Training on Partially-Observed Data for Open Information Extraction](https://mlsanity.com/paper/2301.06841/) (2023)

## Built on by (most cited first, 8 of 8)

- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions](https://mlsanity.com/paper/2506.11111/) (2025)
- [Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models](https://mlsanity.com/paper/2406.02148/) (2024)
- [Mastering the Task of Open Information Extraction with Large Language Models and Consistent Reasoning Environment](https://mlsanity.com/paper/2310.10590/) (2023)
- [Relation Extraction or Pattern Matching? Unravelling the Generalisation Limits of Language Models for Biographical RE](https://mlsanity.com/paper/2505.12533/) (2025)
- [Enhancing the Preference Extractor in Multi-turn Dialogues: From Annotating Disasters to Accurate Preference Extraction](https://mlsanity.com/paper/2508.01739/) (2025)
- [Weak Reward Model Transforms Generative Models into Robust Causal Event Extraction Systems](https://mlsanity.com/paper/2406.18245/) (2024)
- [Can AI Extract Antecedent Factors of Human Trust in AI? An Application of Information Extraction for Scientific Literature in Behavioural and Computer Sciences](https://mlsanity.com/paper/2412.11344/) (2024)

## Abstract (arXiv)

The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior evaluation benchmarks have been devoted to validating pairwise matching correctness, ignoring the crucial measurement of robustness. In this paper, we present the first benchmark that simulates the evaluation of open information extraction models in the real world, where the syntactic and expressive distributions under the same knowledge meaning may drift variously. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique that consists of sentences with structured knowledge of the same meaning but with different syntactic and expressive forms. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques. We perform experiments on typical models published in the last decade as well as a popular large language model, the results show that the existing successful models exhibit a frustrating degradation, with a maximum drop of 23.43 F1 score. Our resources and code are available at https://github.com/qijimrc/ROBUST.