# CrossRE: A Cross-Domain Dataset for Relation Extraction

arXiv: https://arxiv.org/abs/2210.09345 · HTML: https://mlsanity.com/paper/2210.09345/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://mlsanity.com/paper/1601.00770/) (2016)
- [CrossNER: Evaluating Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2012.04373/) (2020)
- [FewRel 2.0: Towards More Challenging Few-Shot Relation Classification](https://mlsanity.com/paper/1910.07124/) (2019)
- [Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders](https://mlsanity.com/paper/2010.03851/) (2020)
- [SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals](https://mlsanity.com/paper/1911.10422/) (2019)
- [Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes](https://mlsanity.com/paper/2104.08481/) (2021)
- [What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification](https://mlsanity.com/paper/2204.13516/) (2022)
- [Why only Micro-F1? Class Weighting of Measures for Relation Classification](https://mlsanity.com/paper/2205.09460/) (2022)

## Built on by (most cited first, 21 of 21)

- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [REFinD: Relation Extraction Financial Dataset](https://mlsanity.com/paper/2305.18322/) (2023)
- [How Good are LLMs at Relation Extraction under Low-Resource Scenario? Comprehensive Evaluation](https://mlsanity.com/paper/2406.11162/) (2024)
- [M-ABSA: A Multilingual Dataset for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2502.11824/) (2025)
- [Beyond Borders: Investigating Cross-Jurisdiction Transfer in Legal Case Summarization](https://mlsanity.com/paper/2403.19317/) (2024)
- [Multi-CrossRE A Multi-Lingual Multi-Domain Dataset for Relation Extraction](https://mlsanity.com/paper/2305.10985/) (2023)
- [M2QA: Multi-domain Multilingual Question Answering](https://mlsanity.com/paper/2407.01091/) (2024)
- [Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations](https://mlsanity.com/paper/2407.04543/) (2024)
- [Silver Syntax Pre-training for Cross-Domain Relation Extraction](https://mlsanity.com/paper/2305.11016/) (2023)
- [CORE: A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation](https://mlsanity.com/paper/2310.12024/) (2023)
- [Can Humans Identify Domains?](https://mlsanity.com/paper/2404.01785/) (2024)
- [Solving Label Variation in Scientific Information Extraction via Multi-Task Learning](https://mlsanity.com/paper/2312.15751/) (2023)
- [Graph-Augmented Relation Extraction Model with LLMs-Generated Support Document](https://mlsanity.com/paper/2410.23452/) (2024)
- [Relation Extraction or Pattern Matching? Unravelling the Generalisation Limits of Language Models for Biographical RE](https://mlsanity.com/paper/2505.12533/) (2025)
- [A systematic review of relation extraction task since the emergence of Transformers](https://mlsanity.com/paper/2511.03610/) (2025)
- [GLiNER-Relex: A Unified Framework for Joint Named Entity Recognition and Relation Extraction](https://mlsanity.com/paper/2605.10108/) (2026)
- [Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferences](https://mlsanity.com/paper/2404.08666/) (2024)
- [How to Encode Domain Information in Relation Classification](https://mlsanity.com/paper/2404.13760/) (2024)
- [Maximizing Relation Extraction Potential: A Data-Centric Study to Unveil Challenges and Opportunities](https://mlsanity.com/paper/2409.04934/) (2024)
- [Semantically Orthogonal Framework for Citation Classification: Disentangling Intent and Content](https://mlsanity.com/paper/2601.05103/) (2026)
- [Calibrated Typed Decisions for Entity, Relation, and Event Extraction](https://mlsanity.com/paper/2610.17113/) (2026)

## Abstract (arXiv)

Relation Extraction (RE) has attracted increasing attention, but current RE evaluation is limited to in-domain evaluation setups. Little is known on how well a RE system fares in challenging, but realistic out-of-distribution evaluation setups. To address this gap, we propose CrossRE, a new, freely-available cross-domain benchmark for RE, which comprises six distinct text domains and includes multi-label annotations. An additional innovation is that we release meta-data collected during annotation, to include explanations and flags of difficult instances. We provide an empirical evaluation with a state-of-the-art model for relation classification. As the meta-data enables us to shed new light on the state-of-the-art model, we provide a comprehensive analysis on the impact of difficult cases and find correlations between model and human annotations. Overall, our empirical investigation highlights the difficulty of cross-domain RE. We release our dataset, to spur more research in this direction.