mlsanityData to 14 Oct 2026

What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification

arXiv 2204.13516 ↗First posted 28 Apr 2022cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Over the last five years, research on Relation Extraction (RE) witnessed extensive progress with many new dataset releases. At the same time, setup clarity has decreased, contributing to increased difficulty of reliable empirical evaluation (Taill'e et al., 2020). In this paper, we provide a comprehensive survey of RE datasets, and revisit the task definition and its adoption by the community. We find that cross-dataset and cross-domain setups are particularly lacking. We present an empirical study on scientific Relation Classification across two datasets. Despite large data overlap, our analysis reveals substantial discrepancies in annotation. Annotation discrepancies strongly impact Relation Classification performance, explaining large drops in cross-dataset evaluations. Variation within further sub-domains exists but impacts Relation Classification only to limited degrees. Overall, our study calls for more rigour in reporting setups in RE and evaluation across multiple test sets.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 43 of its 102 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

43 of 102 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2018Deep contextualized word representationscited by 2,290
  4. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  6. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  7. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 272
  8. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  9. 2019DocRED: A Large-Scale Document-Level Relation Extraction Datasetcited by 239
  10. 2016End-to-End Relation Extraction using LSTMs on Sequences and Tree Structurescited by 161
  11. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 144
  12. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
31 more
  1. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 104
  2. 2017SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publicationscited by 102
  3. 2020Reasoning with Latent Structure Refinement for Document-Level Relation Extractioncited by 97
  4. 2019Entity-Relation Extraction as Multi-Turn Question Answeringcited by 96
  5. 2020TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Taskcited by 96
  6. 2019FewRel 2.0: Towards More Challenging Few-Shot Relation Classificationcited by 94
  7. 2020Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoderscited by 87
  8. 2019Attention Guided Graph Convolutional Networks for Relation Extractioncited by 82
  9. 2020We Need to Talk About Random Splitscited by 71
  10. 2020Dialogue-Based Relation Extractioncited by 68
  11. 2020CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Datasetcited by 66
  12. 2019SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominalscited by 63
  13. 2018Joint entity recognition and relation extraction as a multi-head selection problemcited by 59
  14. 2020Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLPcited by 50
  15. 2017Relation Extraction : A Surveycited by 44
  16. 2021UniRE: A Unified Label Space for Entity Relation Extractioncited by 38
  17. 2018Adversarial training for multi-context joint entity and relation extractioncited by 29
  18. 2020DWIE: an entity-centric dataset for multi-task document-level information extractioncited by 29
  19. 2019Graph Neural Networks with Generated Parameters for Relation Extractioncited by 27
  20. 2021Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemescited by 27
  21. 2021Three Sentences Are All You Need: Local Path Enhanced Document Relation Extractioncited by 16
  22. 2021Revisiting the Negative Data of Distantly Supervised Relation Extractioncited by 15
  23. 2021Entity Concept-enhanced Few-shot Relation Extractioncited by 14
  24. 2021From Discourse to Narrative: Knowledge Projection for Event Relation Extractioncited by 14
  25. 2015Improved Relation Extraction with Feature-Rich Compositional Embedding Modelscited by 12
  26. 2019Exploiting Entity BIO Tag Embeddings and Multi-task Learning for Relation Extraction with Imbalanced Datacited by 11
  27. 2021SENT: Sentence-level Distant Relation Extraction via Negative Trainingcited by 10
  28. 2019A Walk-based Model on Entity Graphs for Relation Extractioncited by 8
  29. 2020Relation Extraction with Explanationcited by 7
  30. 2020In Layman's Terms: Semi-Open Relation Extraction from Scientific Textscited by 6
  31. 2020Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observationscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 3; 2025: 5; 2026: 2.

17 papers.

Sort:

  1. 2023A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontierscited by 22
  2. 2022CrossRE: A Cross-Domain Dataset for Relation Extractioncited by 21
  3. 2025Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judgecited by 13
  4. 2024Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Promptingcited by 10
  5. 2023REDrm FM: a Filtered and Multilingual Relation Extraction Datasetcited by 6
  6. 2023Multi-CrossRE A Multi-Lingual Multi-Domain Dataset for Relation Extractioncited by 5
  7. 2023A Comprehensive Survey of Document-level Relation Extraction (2016-2023)cited by 4
  8. 2024Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluationcited by 2
  9. 2023Solving Label Variation in Scientific Information Extraction via Multi-Task Learningcited by 1
  10. 2025Relation Extraction or Pattern Matching? Unravelling the Generalisation Limits of Language Models for Biographical REcited by 1
  11. 2025A systematic review of relation extraction task since the emergence of Transformerscited by 1
  12. 2023Mining Healthcare Procurement Data Using Text Mining and Natural Language Processing -- Reflection From An Industrial Projectcited by 0
5 more
  1. 2024Maximizing Relation Extraction Potential: A Data-Centric Study to Unveil Challenges and Opportunitiescited by 0
  2. 2025Error-Aware Curriculum Learning for Biomedical Relation Classificationcited by 0
  3. 2025Transformer Enhanced Relation Classification: A Comparative Analysis of Contextuality, Data Efficiency and Sequence Complexitycited by 0
  4. 2026A Study on Building Efficient Zero-Shot Relation Extraction Modelscited by 0
  5. 2026Prompting Is All You Need: Multi-view Prompting Large Language Models for Aspect-Based Sentiment Analysiscited by 0