# Automated Mining of Leaderboards for Empirical AI Research

arXiv: https://arxiv.org/abs/2109.13089 · HTML: https://mlsanity.com/paper/2109.13089/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction](https://mlsanity.com/paper/1906.09317/) (2019)
- [The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures](https://mlsanity.com/paper/1905.06939/) (2019)
- [AxCell: Automatic Extraction of Results from Machine Learning Papers](https://mlsanity.com/paper/2004.14356/) (2020)
- [TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics](https://mlsanity.com/paper/2101.10273/) (2021)
- [An Annotated Corpus for Machine Reading of Instructions in Wet Lab Protocols](https://mlsanity.com/paper/1805.00195/) (2018)
- [SemEval-2021 Task 11: NLPContributionGraph -- Structuring Scholarly NLP Contributions for a Research Knowledge Graph](https://mlsanity.com/paper/2106.07385/) (2021)
- [Annotating and Extracting Synthesis Process of All-Solid-State Batteries from Scientific Literature](https://mlsanity.com/paper/2002.07339/) (2020)
- [Domain-independent Extraction of Scientific Concepts from Research Articles](https://mlsanity.com/paper/2001.03067/) (2020)
- [The STEM-ECR Dataset: Grounding Scientific Entity References in STEM Scholarly Content to Authoritative Encyclopedic and Lexicographic Sources](https://mlsanity.com/paper/2003.01006/) (2020)
- [TabVec: Table Vectors for Classification of Web Tables](https://mlsanity.com/paper/1802.06290/) (2018)
- [A framework for information extraction from tables in biomedical literature](https://mlsanity.com/paper/1902.10031/) (2019)
- [Improving Scholarly Knowledge Representation: Evaluating BERT-based Models for Scientific Relation Classification](https://mlsanity.com/paper/2004.06153/) (2020)
- [Representing Semantified Biological Assays in the Open Research Knowledge Graph](https://mlsanity.com/paper/2009.07642/) (2020)
- [SciBERT-based Semantification of Bioassays in the Open Research Knowledge Graph](https://mlsanity.com/paper/2009.08801/) (2020)
- [Sentence, Phrase, and Triple Annotations to Build a Knowledge Graph of Natural Language Processing Contributions -- A Trial Dataset](https://mlsanity.com/paper/2010.04388/) (2020)

## Built on by (most cited first, 16 of 16)

- [GSAP-NER: A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets](https://mlsanity.com/paper/2311.09860/) (2023)
- [Large Language Models for Scientific Information Extraction: An Empirical Study for Virology](https://mlsanity.com/paper/2401.10040/) (2024)
- [ORKG-Leaderboards: A Systematic Workflow for Mining Leaderboards as a Knowledge Graph](https://mlsanity.com/paper/2305.11068/) (2023)
- [Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards](https://mlsanity.com/paper/2409.12656/) (2024)
- [Computer Science Named Entity Recognition in the Open Research Knowledge Graph](https://mlsanity.com/paper/2203.14579/) (2022)
- [Zero-shot Entailment of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2303.16835/) (2023)
- [Effective Context Selection in LLM-based Leaderboard Generation: An Empirical Study](https://mlsanity.com/paper/2407.02409/) (2024)
- [On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboards](https://mlsanity.com/paper/2407.04065/) (2024)
- [Mining for Species, Locations, Habitats, and Ecosystems from Scientific Papers in Invasion Biology: A Large-Scale Exploratory Study with Large Language Models](https://mlsanity.com/paper/2501.18287/) (2025)
- [A Position Paper on the Automatic Generation of Machine Learning Leaderboards](https://mlsanity.com/paper/2505.17465/) (2025)
- [All Data on the Table: Novel Dataset and Benchmark for Cross-Modality Scientific Information Extraction](https://mlsanity.com/paper/2311.08189/) (2023)
- [LEGOBench: Scientific Leaderboard Generation Benchmark](https://mlsanity.com/paper/2401.06233/) (2024)
- [Clustering Semantic Predicates in the Open Research Knowledge Graph](https://mlsanity.com/paper/2210.02034/) (2022)
- [Overview of STEM Science as Process, Method, Material, and Data Named Entities](https://mlsanity.com/paper/2205.11863/) (2022)
- [Enhancing AI Research Paper Analysis: Methodology Component Extraction using Factored Transformer-based Sequence Modeling Approach](https://mlsanity.com/paper/2311.03401/) (2023)
- [MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments](https://mlsanity.com/paper/2601.22420/) (2026)

## Abstract (arXiv)

With the rapid growth of research publications, empowering scientists to keep oversight over the scientific progress is of paramount importance. In this regard, the Leaderboards facet of information organization provides an overview on the state-of-the-art by aggregating empirical results from various studies addressing the same research challenge. Crowdsourcing efforts like PapersWithCode among others are devoted to the construction of Leaderboards predominantly for various subdomains in Artificial Intelligence. Leaderboards provide machine-readable scholarly knowledge that has proven to be directly useful for scientists to keep track of research progress. The construction of Leaderboards could be greatly expedited with automated text mining. This study presents a comprehensive approach for generating Leaderboards for knowledge-graph-based scholarly information organization. Specifically, we investigate the problem of automated Leaderboard construction using state-of-the-art transformer models, viz. Bert, SciBert, and XLNet. Our analysis reveals an optimal approach that significantly outperforms existing baselines for the task with evaluation scores above 90% in F1. This, in turn, offers new state-of-the-art results for Leaderboard extraction. As a result, a vast share of empirical AI research can be organized in the next-generation digital libraries as knowledge graphs.