# SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents

arXiv: https://arxiv.org/abs/2410.21155 · HTML: https://mlsanity.com/paper/2410.21155/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](https://mlsanity.com/paper/2303.08559/) (2023)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [Revisiting Relation Extraction in the era of Large Language Models](https://mlsanity.com/paper/2305.05003/) (2023)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Bases](https://mlsanity.com/paper/2210.12925/) (2022)
- [COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation](https://mlsanity.com/paper/2007.00576/) (2020)
- [TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics](https://mlsanity.com/paper/2101.10273/) (2021)
- [Noisy-Labeled NER with Confidence Estimation](https://mlsanity.com/paper/2104.04318/) (2021)
- [SciNLI: A Corpus for Natural Language Inference on Scientific Text](https://mlsanity.com/paper/2203.06728/) (2022)
- [FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding](https://mlsanity.com/paper/2407.05183/) (2024)
- [Scientific Information Extraction with Semi-supervised Neural Tagging](https://mlsanity.com/paper/1708.06075/) (2017)
- [CitationIE: Leveraging the Citation Graph for Scientific Information Extraction](https://mlsanity.com/paper/2106.01560/) (2021)
- [DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions](https://mlsanity.com/paper/2305.16636/) (2023)
- [TempEL: Linking Dynamically Evolving and Newly Emerging Entities](https://mlsanity.com/paper/2302.02500/) (2023)
- [GSAP-NER: A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets](https://mlsanity.com/paper/2311.09860/) (2023)
- [DMDD: A Large-Scale Dataset for Dataset Mentions Detection](https://mlsanity.com/paper/2305.11779/) (2023)
- [Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks](https://mlsanity.com/paper/2310.17238/) (2023)
- [SciDMT: A Large-Scale Corpus for Detecting Scientific Mentions](https://mlsanity.com/paper/2406.14756/) (2024)
- [Open Information Extraction on Scientific Text: An Evaluation](https://mlsanity.com/paper/1802.05574/) (2018)
- [Boosting Entity Mention Detection for Targetted Twitter Streams with Global Contextual Embeddings](https://mlsanity.com/paper/2201.11885/) (2022)

## Built on by (most cited first, 21 of 21)

- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [SciNLP: A Domain-Specific Benchmark for Full-Text Scientific Entity and Relation Extraction in NLP](https://mlsanity.com/paper/2509.07801/) (2025)
- [Multi-Disciplinary Dataset Discovery from Citation-Verified Literature Contexts](https://mlsanity.com/paper/2601.05099/) (2026)
- [DynClean: Training Dynamics-based Label Cleaning for Distantly-Supervised Named Entity Recognition](https://mlsanity.com/paper/2504.04616/) (2025)
- [Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R2)GRPO](https://mlsanity.com/paper/2505.22068/) (2025)
- [Efficient Scientific Full Text Classification: The Case of EICAT Impact Assessments](https://mlsanity.com/paper/2502.06551/) (2025)
- [SciEvent: Benchmarking Multi-domain Scientific Event Extraction](https://mlsanity.com/paper/2509.15620/) (2025)
- [BCL: Bayesian In-Context Learning Framework for Information Extraction](https://mlsanity.com/paper/2606.18620/) (2026)
- [Building Agent Harnesses for Scientific Curation from Multimodal Sources](https://mlsanity.com/paper/2606.21005/) (2026)
- [LA-RL: Label-Aware Self-Reflection for Reinforcement Learning in Information Extraction](https://mlsanity.com/paper/2607.23420/) (2026)
- [EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge](https://mlsanity.com/paper/2507.03617/) (2025)
- [Querying Climate Knowledge: Semantic Retrieval for Scientific Discovery](https://mlsanity.com/paper/2509.10087/) (2025)
- [BioPIE: A Biomedical Protocol Information Extraction Dataset for Experiment Understanding](https://mlsanity.com/paper/2601.04524/) (2026)
- [Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis](https://mlsanity.com/paper/2602.10881/) (2026)
- [HGNet: Scalable Foundation Model for Automated Knowledge Graph Generation from Scientific Literature](https://mlsanity.com/paper/2603.23136/) (2026)
- [SSDAU: Structured Semantic Data Augmentation for Joint Entity and Relation Extraction](https://mlsanity.com/paper/2605.23440/) (2026)
- [Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition](https://mlsanity.com/paper/2606.21890/) (2026)
- [SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text](https://mlsanity.com/paper/2607.21610/) (2026)
- [MUSE: A Full-Text Cross-Domain Knowledge Base of Scientific Problems, Solutions, and Rationales](https://mlsanity.com/paper/2608.10974/) (2026)
- [SCoR: A Hierarchical Framework for Forecasting Relations Between Scientific Concepts](https://mlsanity.com/paper/2609.22174/) (2026)
- [Calibrated Typed Decisions for Entity, Relation, and Event Extraction](https://mlsanity.com/paper/2610.17113/) (2026)

## Abstract (arXiv)

Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data (entities and relations). Several datasets have been proposed for training and validating SciIE models. However, due to the high complexity and cost of annotating scientific texts, those datasets restrict their annotations to specific parts of paper, such as abstracts, resulting in the loss of diverse entity mentions and relations in context. In this paper, we release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations. To capture the intricate use and interactions among entities in full texts, our dataset contains a fine-grained tag set for relations. Additionally, we provide an out-of-distribution test set to offer a more realistic evaluation. We conduct comprehensive experiments, including state-of-the-art supervised models and our proposed LLM-based baselines, and highlight the challenges presented by our dataset, encouraging the development of innovative models to further the field of SciIE.