# TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics

arXiv: https://arxiv.org/abs/2101.10273 · HTML: https://mlsanity.com/paper/2101.10273/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 25 Jan 2021.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction](https://mlsanity.com/paper/1906.09317/) (2019)
- [A Summarization System for Scientific Documents](https://mlsanity.com/paper/1908.11152/) (2019)

## Built on by (most cited first, 30 of 30)

- [End-to-End NLP Knowledge Graph Construction](https://mlsanity.com/paper/2106.01167/) (2021)
- [Automated scholarly paper review: Concepts, technologies, and challenges](https://mlsanity.com/paper/2111.07533/) (2021)
- [SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents](https://mlsanity.com/paper/2410.21155/) (2024)
- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [Automated Mining of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2109.13089/) (2021)
- [A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?](https://mlsanity.com/paper/2305.12920/) (2023)
- [GSAP-NER: A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets](https://mlsanity.com/paper/2311.09860/) (2023)
- [The Nature of NLP: Analyzing Contributions in NLP Papers](https://mlsanity.com/paper/2409.19505/) (2024)
- [DMDD: A Large-Scale Dataset for Dataset Mentions Detection](https://mlsanity.com/paper/2305.11779/) (2023)
- [Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards](https://mlsanity.com/paper/2409.12656/) (2024)
- [ACLM: A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER](https://mlsanity.com/paper/2306.00928/) (2023)
- [Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End](https://mlsanity.com/paper/2212.10522/) (2022)
- [Zero-shot Entailment of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2303.16835/) (2023)
- [SciDMT: A Large-Scale Corpus for Detecting Scientific Mentions](https://mlsanity.com/paper/2406.14756/) (2024)
- [A Position Paper on the Automatic Generation of Machine Learning Leaderboards](https://mlsanity.com/paper/2505.17465/) (2025)
- [NASA Science Mission Directorate Knowledge Graph Discovery](https://mlsanity.com/paper/2303.10871/) (2023)
- [All Data on the Table: Novel Dataset and Benchmark for Cross-Modality Scientific Information Extraction](https://mlsanity.com/paper/2311.08189/) (2023)
- [Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey](https://mlsanity.com/paper/2503.09701/) (2025)
- [TaxoAlign: Scholarly Taxonomy Generation Using Language Models](https://mlsanity.com/paper/2510.17263/) (2025)
- [Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective](https://mlsanity.com/paper/2606.29836/) (2026)
- [RAAMove: A Corpus for Analyzing Moves in Research Article Abstracts](https://mlsanity.com/paper/2403.15872/) (2024)
- [Quantifying the Knowledge Proximity Between Academic and Industry Research: An Entity and Semantic Perspective](https://mlsanity.com/paper/2602.05211/) (2026)
- [ClaimFlow: Tracing the Evolution of Scientific Claims in NLP](https://mlsanity.com/paper/2603.16073/) (2026)
- [Hierarchical Transformer Model for Scientific Named Entity Recognition](https://mlsanity.com/paper/2203.14710/) (2022)
- [Enhancing AI Research Paper Analysis: Methodology Component Extraction using Factored Transformer-based Sequence Modeling Approach](https://mlsanity.com/paper/2311.03401/) (2023)
- [Scaling Automatic Extraction of Pseudocode](https://mlsanity.com/paper/2406.04635/) (2024)
- [Enhancing Academic Paper Recommendations Using Fine-Grained Knowledge Entities and Multifaceted Document Embeddings](https://mlsanity.com/paper/2601.19513/) (2026)
- [MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments](https://mlsanity.com/paper/2601.22420/) (2026)
- [Enhancing Scientific Named Entity Recognition via Large Language Models: A Type-driven Multi-task Learning Approach](https://mlsanity.com/paper/2608.08636/) (2026)
- [ShadowMiner v1 - An Experience Report on Implementing and Measuring a Problem-and-Hypothesis Discovery Engine](https://mlsanity.com/paper/2610.04339/) (2026)

## Abstract (arXiv)

Tasks, Datasets and Evaluation Metrics are important concepts for understanding experimental scientific papers. However, most previous work on information extraction for scientific literature mainly focuses on the abstracts only, and does not treat datasets as a separate type of entity (Zadeh and Schumann, 2016; Luan et al., 2018). In this paper, we present a new corpus that contains domain expert annotations for Task (T), Dataset (D), Metric (M) entities on 2,000 sentences extracted from NLP papers. We report experiment results on TDM extraction using a simple data augmentation strategy and apply our tagger to around 30,000 NLP papers from the ACL Anthology. The corpus is made publicly available to the community for fostering research on scientific publication summarization (Erera et al., 2019) and knowledge discovery.