mlsanityData to 7 Oct 2026

Good Data, Large Data, or No Data? Comparing Three Approaches in Developing Research Aspect Classifiers for Biomedical Papers

arXiv 2306.04820 ↗First posted 7 Jun 2023cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

The rapid growth of scientific publications, particularly during the COVID-19 pandemic, emphasizes the need for tools to help researchers efficiently comprehend the latest advancements. One essential part of understanding scientific literature is research aspect classification, which categorizes sentences in abstracts to Background, Purpose, Method, and Finding. In this study, we investigate the impact of different datasets on model performance for the crowd-annotated CODA-19 research aspect classification task. Specifically, we explore the potential benefits of using the large, automatically curated PubMed 200K RCT dataset and evaluate the effectiveness of large language models (LLMs), such as LLaMA, GPT-3, ChatGPT, and GPT-4. Our results indicate that using the PubMed 200K RCT dataset does not improve performance for the CODA-19 task. We also observe that while GPT-4 performs well, it does not outperform the SciBERT model fine-tuned on the CODA-19 dataset, emphasizing the importance of a dedicated and task-aligned datasets dataset for the target task. Our code is available at https://github.com/Crowd-AI-Lab/CODA-19-exp.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 18 of its 35 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

18 of 35 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  9. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  10. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 123
  11. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  12. 2020Scientific Claim Verification with VERT5ERINIcited by 47
6 more
  1. 2021GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domaincited by 36
  2. 2021Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documentscited by 33
  3. 2020SciSight: Combining faceted navigation and research group detection for COVID-19 exploratory scientific searchcited by 16
  4. 2021A Search Engine for Discovery of Scientific Challenges and Directionscited by 12
  5. 2020Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learnedcited by 10
  6. 2020Exploration and Discovery of the COVID-19 Literature through Semantic Visualizationcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  2. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38