mlsanityData to 14 Oct 2026

Instruction Finetuning for Leaderboard Generation from Empirical AI Research

arXiv 2408.10141 ↗First posted 19 Aug 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This study demonstrates the application of instruction finetuning of pretrained Large Language Models (LLMs) to automate the generation of AI research leaderboards, extracting (Task, Dataset, Metric, Score) quadruples from articles. It aims to streamline the dissemination of advancements in AI research by transitioning from traditional, manual community curation, or otherwise taxonomy-constrained natural language inference (NLI) models, to an automated, generative LLM-based approach. Utilizing the FLAN-T5 model, this research enhances LLMs' adaptability and reliability in information extraction, offering a novel method for structured knowledge representation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

22 of 34 references.

References · 22

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  8. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  9. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  10. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  11. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  12. 2023InstructUIE: Multi-task Instruction Tuning for Unified Information Extractioncited by 170
10 more
  1. 2020SciREX: A Challenge Dataset for Document-Level Information Extractioncited by 113
  2. 2017SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publicationscited by 102
  3. 2020Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databasescited by 75
  4. 2019Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Constructioncited by 50
  5. 2020AxCell: Automatic Extraction of Results from Machine Learning Paperscited by 48
  6. 2021End-to-End NLP Knowledge Graph Constructioncited by 21
  7. 2023ORKG-Leaderboards: A Systematic Workflow for Mining Leaderboards as a Knowledge Graphcited by 11
  8. 2022Computer Science Named Entity Recognition in the Open Research Knowledge Graphcited by 9
  9. 2023Zero-shot Entailment of Leaderboards for Empirical AI Researchcited by 6
  10. 2021Pattern-based Acquisition of Scientific Entities from Scholarly Article Titlescited by 4

Built on by

Not cited yet.