# What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification

arXiv: https://arxiv.org/abs/2204.13516 · HTML: https://mlsanity.com/paper/2204.13516/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 102 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://mlsanity.com/paper/1601.00770/) (2016)
- [SParC: Cross-Domain Semantic Parsing in Context](https://mlsanity.com/paper/1906.02285/) (2019)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [CrossNER: Evaluating Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2012.04373/) (2020)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [Reasoning with Latent Structure Refinement for Document-Level Relation Extraction](https://mlsanity.com/paper/2005.06312/) (2020)
- [Entity-Relation Extraction as Multi-Turn Question Answering](https://mlsanity.com/paper/1905.05529/) (2019)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [FewRel 2.0: Towards More Challenging Few-Shot Relation Classification](https://mlsanity.com/paper/1910.07124/) (2019)
- [Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders](https://mlsanity.com/paper/2010.03851/) (2020)
- [Attention Guided Graph Convolutional Networks for Relation Extraction](https://mlsanity.com/paper/1906.07510/) (2019)
- [We Need to Talk About Random Splits](https://mlsanity.com/paper/2005.00636/) (2020)
- [Dialogue-Based Relation Extraction](https://mlsanity.com/paper/2004.08056/) (2020)
- [CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset](https://mlsanity.com/paper/2002.11893/) (2020)
- [SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals](https://mlsanity.com/paper/1911.10422/) (2019)
- [Joint entity recognition and relation extraction as a multi-head selection problem](https://mlsanity.com/paper/1804.07847/) (2018)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [Relation Extraction : A Survey](https://mlsanity.com/paper/1712.05191/) (2017)
- [UniRE: A Unified Label Space for Entity Relation Extraction](https://mlsanity.com/paper/2107.04292/) (2021)
- [Adversarial training for multi-context joint entity and relation extraction](https://mlsanity.com/paper/1808.06876/) (2018)
- [DWIE: an entity-centric dataset for multi-task document-level information extraction](https://mlsanity.com/paper/2009.12626/) (2020)

## Built on by (most cited first, 17 of 17)

- [A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers](https://mlsanity.com/paper/2306.02051/) (2023)
- [CrossRE: A Cross-Domain Dataset for Relation Extraction](https://mlsanity.com/paper/2210.09345/) (2022)
- [Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge](https://mlsanity.com/paper/2506.00777/) (2025)
- [Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Prompting](https://mlsanity.com/paper/2410.01154/) (2024)
- [REDrm FM: a Filtered and Multilingual Relation Extraction Dataset](https://mlsanity.com/paper/2306.09802/) (2023)
- [Multi-CrossRE A Multi-Lingual Multi-Domain Dataset for Relation Extraction](https://mlsanity.com/paper/2305.10985/) (2023)
- [A Comprehensive Survey of Document-level Relation Extraction (2016-2023)](https://mlsanity.com/paper/2309.16396/) (2023)
- [Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation](https://mlsanity.com/paper/2404.04445/) (2024)
- [Solving Label Variation in Scientific Information Extraction via Multi-Task Learning](https://mlsanity.com/paper/2312.15751/) (2023)
- [Relation Extraction or Pattern Matching? Unravelling the Generalisation Limits of Language Models for Biographical RE](https://mlsanity.com/paper/2505.12533/) (2025)
- [A systematic review of relation extraction task since the emergence of Transformers](https://mlsanity.com/paper/2511.03610/) (2025)
- [Mining Healthcare Procurement Data Using Text Mining and Natural Language Processing -- Reflection From An Industrial Project](https://mlsanity.com/paper/2301.03458/) (2023)
- [Maximizing Relation Extraction Potential: A Data-Centric Study to Unveil Challenges and Opportunities](https://mlsanity.com/paper/2409.04934/) (2024)
- [Error-Aware Curriculum Learning for Biomedical Relation Classification](https://mlsanity.com/paper/2507.14374/) (2025)
- [Transformer Enhanced Relation Classification: A Comparative Analysis of Contextuality, Data Efficiency and Sequence Complexity](https://mlsanity.com/paper/2509.11374/) (2025)
- [A Study on Building Efficient Zero-Shot Relation Extraction Models](https://mlsanity.com/paper/2603.01266/) (2026)
- [Prompting Is All You Need: Multi-view Prompting Large Language Models for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2605.28058/) (2026)

## Abstract (arXiv)

Over the last five years, research on Relation Extraction (RE) witnessed extensive progress with many new dataset releases. At the same time, setup clarity has decreased, contributing to increased difficulty of reliable empirical evaluation (Taill'e et al., 2020). In this paper, we provide a comprehensive survey of RE datasets, and revisit the task definition and its adoption by the community. We find that cross-dataset and cross-domain setups are particularly lacking. We present an empirical study on scientific Relation Classification across two datasets. Despite large data overlap, our analysis reveals substantial discrepancies in annotation. Annotation discrepancies strongly impact Relation Classification performance, explaining large drops in cross-dataset evaluations. Variation within further sub-domains exists but impacts Relation Classification only to limited degrees. Overall, our study calls for more rigour in reporting setups in RE and evaluation across multiple test sets.