mlsanityData to 14 Oct 2026

Exploring the Latest LLMs for Leaderboard Extraction

arXiv 2406.04383 ↗First posted 6 Jun 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The rapid advancements in Large Language Models (LLMs) have opened new avenues for automating complex tasks in AI research. This paper investigates the efficacy of different LLMs-Mistral 7B, Llama-2, GPT-4-Turbo and GPT-4.o in extracting leaderboard information from empirical AI research articles. We explore three types of contextual inputs to the models: DocTAET (Document Title, Abstract, Experimental Setup, and Tabular Information), DocREC (Results, Experiments, and Conclusions), and DocFULL (entire document). Our comprehensive study evaluates the performance of these models in generating (Task, Dataset, Metric, Score) quadruples from research papers. The findings reveal significant insights into the strengths and limitations of each model and context type, providing valuable guidance for future AI research automation efforts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

25 of 33 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
13 more
  1. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  2. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  3. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  4. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  5. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  6. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  7. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  8. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  9. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  10. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  11. 2020Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databasescited by 75
  12. 2019Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Constructioncited by 50
  13. 2024Large Language Models for Scientific Information Extraction: An Empirical Study for Virologycited by 12

Built on by

Not cited yet.