# Automated Early Leaderboard Generation From Comparative Tables

arXiv: https://arxiv.org/abs/1802.04538 · HTML: https://mlsanity.com/paper/1802.04538/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2018.

## Built on by (most cited first, 9 of 9)

- [TabLeX: A Benchmark Dataset for Structure and Content Information Extraction from Scientific Tables](https://mlsanity.com/paper/2105.06400/) (2021)
- [Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards](https://mlsanity.com/paper/2409.12656/) (2024)
- [ICDAR 2021 Competition on Scientific Table Image Recognition to LaTeX](https://mlsanity.com/paper/2105.14426/) (2021)
- [Tables to LaTeX: structure and content extraction from scientific tables](https://mlsanity.com/paper/2210.17246/) (2022)
- [Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs](https://mlsanity.com/paper/2502.18791/) (2025)
- [A Position Paper on the Automatic Generation of Machine Learning Leaderboards](https://mlsanity.com/paper/2505.17465/) (2025)
- [LEGOBench: Scientific Leaderboard Generation Benchmark](https://mlsanity.com/paper/2401.06233/) (2024)
- [Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions](https://mlsanity.com/paper/2509.20645/) (2025)
- [MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments](https://mlsanity.com/paper/2601.22420/) (2026)

## Abstract (arXiv)

A leaderboard is a tabular presentation of performance scores of the best competing techniques that address a specific scientific problem. Manually maintained leaderboards take time to emerge, which induces a latency in performance discovery and meaningful comparison. This can delay dissemination of best practices to non-experts and practitioners. Regarding papers as proxies for techniques, we present a new system to automatically discover and maintain leaderboards in the form of partial orders between papers, based on performance reported therein. In principle, a leaderboard depends on the task, data set, other experimental settings, and the choice of performance metrics. Often there are also tradeoffs between different metrics. Thus, leaderboard discovery is not just a matter of accurately extracting performance numbers and comparing them. In fact, the levels of noise and uncertainty around performance comparisons are so large that reliable traditional extraction is infeasible. We mitigate these challenges by using relatively cleaner, structured parts of the papers, e.g., performance tables. We propose a novel performance improvement graph with papers as nodes, where edges encode noisy performance comparison information extracted from tables. Every individual performance edge is extracted from a table with citations to other papers. These extractions resemble (noisy) outcomes of 'matches' in an incomplete tournament. We propose several approaches to rank papers from these noisy 'match' outcomes. We show that our ranking scheme can reproduce various manually curated leaderboards very well. Using widely-used lists of state-of-the-art papers in 27 areas of Computer Science, we demonstrate that our system produces very reliable rankings.