# LEGOBench: Scientific Leaderboard Generation Benchmark

arXiv: https://arxiv.org/abs/2401.06233 · HTML: https://mlsanity.com/paper/2401.06233/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 24 of 24)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [TableBank: A Benchmark Dataset for Table Detection and Recognition](https://mlsanity.com/paper/1903.01949/) (2019)
- [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings](https://mlsanity.com/paper/2202.06671/) (2022)
- [Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction](https://mlsanity.com/paper/1906.09317/) (2019)
- [AxCell: Automatic Extraction of Results from Machine Learning Papers](https://mlsanity.com/paper/2004.14356/) (2020)
- [Scientific Information Extraction with Semi-supervised Neural Tagging](https://mlsanity.com/paper/1708.06075/) (2017)
- [OAG-BERT: Towards A Unified Backbone Language Model For Academic Knowledge Services](https://mlsanity.com/paper/2103.02410/) (2021)
- [CitationIE: Leveraging the Citation Graph for Scientific Information Extraction](https://mlsanity.com/paper/2106.01560/) (2021)
- [Automated Mining of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2109.13089/) (2021)
- [Automated Early Leaderboard Generation From Comparative Tables](https://mlsanity.com/paper/1802.04538/) (2018)
- [Tables to LaTeX: structure and content extraction from scientific tables](https://mlsanity.com/paper/2210.17246/) (2022)
- [Zero-shot Entailment of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2303.16835/) (2023)
- [Why Did You Not Compare With That? Identifying Papers for Use as Baselines](https://mlsanity.com/paper/2201.08089/) (2022)

## Built on by (most cited first, 2 of 2)

- [SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence](https://mlsanity.com/paper/2512.22334/) (2025)
- [League: Leaderboard Generation on Demand](https://mlsanity.com/paper/2502.18209/) (2025)

## Abstract (arXiv)

The ever-increasing volume of paper submissions makes it difficult to stay informed about the latest state-of-the-art research. To address this challenge, we introduce LEGOBench, a benchmark for evaluating systems that generate scientific leaderboards. LEGOBench is curated from 22 years of preprint submission data on arXiv and more than 11k machine learning leaderboards on the PapersWithCode portal. We present four graph-based and two language model-based leaderboard generation task configurations. We evaluate popular encoder-only scientific language models as well as decoder-only large language models across these task configurations. State-of-the-art models showcase significant performance gaps in automatic leaderboard generation on LEGOBench. The code is available on GitHub ( https://github.com/lingo-iitgn/LEGOBench ) and the dataset is hosted on OSF ( https://osf.io/9v2py/?view_only=6f91b0b510df498ba01595f8f278f94c ).