# Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction

arXiv: https://arxiv.org/abs/2109.12008 · HTML: https://mlsanity.com/paper/2109.12008/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 24 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Span-based Joint Entity and Relation Extraction with Transformer Pre-training](https://mlsanity.com/paper/1909.07755/) (2019)
- [Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders](https://mlsanity.com/paper/2010.03851/) (2020)
- [Learning from Context or Names? An Empirical Study on Neural Relation Extraction](https://mlsanity.com/paper/2010.01923/) (2020)
- [Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study](https://mlsanity.com/paper/2001.03844/) (2020)
- [Adversarial training for multi-context joint entity and relation extraction](https://mlsanity.com/paper/1808.06876/) (2018)
- [Interpretable Multi-dataset Evaluation for Named Entity Recognition](https://mlsanity.com/paper/2011.06854/) (2020)
- [Generalisation in Named Entity Recognition: A Quantitative Analysis](https://mlsanity.com/paper/1701.02877/) (2017)
- [Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data](https://mlsanity.com/paper/2010.03656/) (2020)
- [Contextualized Embeddings in Named-Entity Recognition: An Empirical Study on Generalization](https://mlsanity.com/paper/2001.08053/) (2020)
- [Contextual Embeddings: When Are They Worth It?](https://mlsanity.com/paper/2005.09117/) (2020)
- [Lexical Features in Coreference Resolution: To be Used With Caution](https://mlsanity.com/paper/1704.06779/) (2017)

## Built on by (most cited first, 2 of 2)

- [MedDistant19: Towards an Accurate Benchmark for Broad-Coverage Biomedical Relation Extraction](https://mlsanity.com/paper/2204.04779/) (2022)
- [REXEL: An End-to-end Model for Document-Level Relation Extraction and Entity Linking](https://mlsanity.com/paper/2404.12788/) (2024)

## Abstract (arXiv)

State-of-the-art NLP models can adopt shallow heuristics that limit their generalization capability (McCoy et al., 2019). Such heuristics include lexical overlap with the training set in Named-Entity Recognition (Taill'e et al., 2020) and Event or Type heuristics in Relation Extraction (Rosenman et al., 2020). In the more realistic end-to-end RE setting, we can expect yet another heuristic: the mere retention of training relation triples. In this paper, we propose several experiments confirming that retention of known facts is a key factor of performance on standard benchmarks. Furthermore, one experiment suggests that a pipeline model able to use intermediate type representations is less prone to over-rely on retention.