# SciREX: A Challenge Dataset for Document-Level Information Extraction

arXiv: https://arxiv.org/abs/2005.00512 · HTML: https://mlsanity.com/paper/2005.00512/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 30 of 113)

- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [MS2: Multi-Document Summarization of Medical Studies](https://mlsanity.com/paper/2104.06486/) (2021)
- [UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition](https://mlsanity.com/paper/2308.03279/) (2023)
- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text](https://mlsanity.com/paper/2308.02357/) (2023)
- [Document-level Entity-based Extraction as Template Generation](https://mlsanity.com/paper/2109.04901/) (2021)
- [TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics](https://mlsanity.com/paper/2101.10273/) (2021)
- [A sequence-to-sequence approach for document-level relation extraction](https://mlsanity.com/paper/2204.01098/) (2022)
- [LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs](https://mlsanity.com/paper/2408.08656/) (2024)
- [F-coref: Fast, Accurate and Easy to Use Coreference Resolution](https://mlsanity.com/paper/2209.04280/) (2022)
- [A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers](https://mlsanity.com/paper/2306.02051/) (2023)
- [End-to-End NLP Knowledge Graph Construction](https://mlsanity.com/paper/2106.01167/) (2021)
- [SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents](https://mlsanity.com/paper/2410.21155/) (2024)
- [CitationIE: Leveraging the Citation Graph for Scientific Information Extraction](https://mlsanity.com/paper/2106.01560/) (2021)
- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [Automated Mining of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2109.13089/) (2021)
- [DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions](https://mlsanity.com/paper/2305.16636/) (2023)
- [READoc: A Unified Benchmark for Realistic Document Structured Extraction](https://mlsanity.com/paper/2409.05137/) (2024)
- [SciCo: Hierarchical Cross-Document Coreference for Scientific Concepts](https://mlsanity.com/paper/2104.08809/) (2021)
- [PolyIE: A Dataset of Information Extraction from Polymer Material Scientific Literature](https://mlsanity.com/paper/2311.07715/) (2023)
- [Deep Neural Approaches to Relation Triplets Extraction: A Comprehensive Survey](https://mlsanity.com/paper/2103.16929/) (2021)
- [A Search Engine for Discovery of Scientific Challenges and Directions](https://mlsanity.com/paper/2108.13751/) (2021)
- [GSAP-NER: A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets](https://mlsanity.com/paper/2311.09860/) (2023)
- [Few-Shot Document-Level Relation Extraction](https://mlsanity.com/paper/2205.02048/) (2022)
- [Iterative Document-level Information Extraction via Imitation Learning](https://mlsanity.com/paper/2210.06600/) (2022)
- [ORKG-Leaderboards: A Systematic Workflow for Mining Leaderboards as a Knowledge Graph](https://mlsanity.com/paper/2305.11068/) (2023)
- [DMDD: A Large-Scale Dataset for Dataset Mentions Detection](https://mlsanity.com/paper/2305.11779/) (2023)
- [Text2NKG: Fine-Grained N-ary Relation Extraction for N-ary relational Knowledge Graph Construction](https://mlsanity.com/paper/2310.05185/) (2023)

## Abstract (arXiv)

Extracting information from full documents is an important problem in many domains, but most previous work focus on identifying relationships within a sentence or a paragraph. It is challenging to create a large-scale information extraction (IE) dataset at the document level since it requires an understanding of the whole document to annotate entities and their document-level relationships that usually span beyond sentences or even sections. In this paper, we introduce SciREX, a document level IE dataset that encompasses multiple IE tasks, including salient entity identification and document level N-ary relation identification from scientific articles. We annotate our dataset by integrating automatic and human annotations, leveraging existing scientific knowledge resources. We develop a neural model as a strong baseline that extends previous state-of-the-art IE models to document-level IE. Analyzing the model performance shows a significant gap between human performance and current baselines, inviting the community to use our dataset as a challenge to develop document-level IE models. Our data and code are publicly available at https://github.com/allenai/SciREX