# Effective Context Selection in LLM-based Leaderboard Generation: An Empirical Study

arXiv: https://arxiv.org/abs/2407.02409 · HTML: https://mlsanity.com/paper/2407.02409/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 9 of 9)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases](https://mlsanity.com/paper/2012.07675/) (2020)
- [AxCell: Automatic Extraction of Results from Machine Learning Papers](https://mlsanity.com/paper/2004.14356/) (2020)
- [Automated Mining of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2109.13089/) (2021)
- [Zero-shot Entailment of Leaderboards for Empirical AI Research](https://mlsanity.com/paper/2303.16835/) (2023)

## Built on by (most cited first, 6 of 6)

- [MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering](https://mlsanity.com/paper/2505.24040/) (2025)
- [A Position Paper on the Automatic Generation of Machine Learning Leaderboards](https://mlsanity.com/paper/2505.17465/) (2025)
- [NoteEx: Interactive Visual Context Manipulation for LLM-Assisted Exploratory Data Analysis in Computational Notebooks](https://mlsanity.com/paper/2511.07223/) (2025)
- [Identifying and Addressing User-level Security Concerns in Smart Homes Using "Smaller" LLMs](https://mlsanity.com/paper/2509.19485/) (2025)
- [MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments](https://mlsanity.com/paper/2601.22420/) (2026)
- [Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis](https://mlsanity.com/paper/2602.10881/) (2026)

## Abstract (arXiv)

This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.