# Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction

arXiv: https://arxiv.org/abs/1906.09317 · HTML: https://mlsanity.com/paper/1906.09317/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2019.

## Built on by (most cited first, 30 of 50)

- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [D2S: Document-to-Slide Generation Via Query-Based Text Summarization](https://mlsanity.com/paper/2105.03664/) (2021)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics](https://mlsanity.com/paper/2101.10273/) (2021)
- [Schema-Driven Information Extraction from Heterogeneous Tables](https://mlsanity.com/paper/2305.14336/) (2023)
- [A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers](https://mlsanity.com/paper/2306.02051/) (2023)
- [End-to-End NLP Knowledge Graph Construction](https://mlsanity.com/paper/2106.01167/) (2021)
- [Automated scholarly paper review: Concepts, technologies, and challenges](https://mlsanity.com/paper/2111.07533/) (2021)
- [CitationIE: Leveraging the Citation Graph for Scientific Information Extraction](https://mlsanity.com/paper/2106.01560/) (2021)
- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [Automated Mining of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2109.13089/) (2021)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?](https://mlsanity.com/paper/2305.12920/) (2023)
- [PolyIE: A Dataset of Information Extraction from Polymer Material Scientific Literature](https://mlsanity.com/paper/2311.07715/) (2023)
- [The Nature of NLP: Analyzing Contributions in NLP Papers](https://mlsanity.com/paper/2409.19505/) (2024)
- [ORKG-Leaderboards: A Systematic Workflow for Mining Leaderboards as a Knowledge Graph](https://mlsanity.com/paper/2305.11068/) (2023)
- [DMDD: A Large-Scale Dataset for Dataset Mentions Detection](https://mlsanity.com/paper/2305.11779/) (2023)
- [ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select](https://mlsanity.com/paper/2210.14427/) (2022)
- [Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards](https://mlsanity.com/paper/2409.12656/) (2024)
- [Computer Science Named Entity Recognition in the Open Research Knowledge Graph](https://mlsanity.com/paper/2203.14579/) (2022)
- [A Dataset for N-ary Relation Extraction of Drug Combinations](https://mlsanity.com/paper/2205.02289/) (2022)
- [Zero-shot Entailment of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2303.16835/) (2023)
- [SciDMT: A Large-Scale Corpus for Detecting Scientific Mentions](https://mlsanity.com/paper/2406.14756/) (2024)
- [Investigating Text Simplification Evaluation](https://mlsanity.com/paper/2107.13662/) (2021)
- [CARE: Extracting Experimental Findings From Clinical Literature](https://mlsanity.com/paper/2311.09736/) (2023)
- [Better Call the Plumber: Orchestrating Dynamic Information Extraction Pipelines](https://mlsanity.com/paper/2102.10966/) (2021)
- [COMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviews](https://mlsanity.com/paper/2108.04366/) (2021)
- [Pattern-based Acquisition of Scientific Entities from Scholarly Article Titles](https://mlsanity.com/paper/2109.00199/) (2021)
- [S2abEL: A Dataset for Entity Linking from Scientific Tables](https://mlsanity.com/paper/2305.00366/) (2023)
- [Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs](https://mlsanity.com/paper/2502.18791/) (2025)

## Abstract (arXiv)

While the fast-paced inception of novel tasks and new datasets helps foster active research in a community towards interesting directions, keeping track of the abundance of research activity in different areas on different datasets is likely to become increasingly difficult. The community could greatly benefit from an automatic system able to summarize scientific results, e.g., in the form of a leaderboard. In this paper we build two datasets and develop a framework (TDMS-IE) aimed at automatically extracting task, dataset, metric and score from NLP papers, towards the automatic construction of leaderboards. Experiments show that our model outperforms several baselines by a large margin. Our model is a first step towards automatic leaderboard construction, e.g., in the NLP domain.