mlsanityData to 7 Oct 2026

Transformer-based Language Models for Factoid Question Answering at BioASQ9b

arXiv 2109.07185 ↗First posted 15 Sep 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this work, we describe our experiments and participating systems in the BioASQ Task 9b Phase B challenge of biomedical question answering. We have focused on finding the ideal answers and investigated multi-task fine-tuning and gradual unfreezing techniques on transformer-based language models. For factoid questions, our ALBERT-based systems ranked first in test batch 1 and fourth in test batch 2. Our DistilBERT systems outperformed the ALBERT variants in test batches 4 and 5 despite having 81% fewer parameters than ALBERT. However, we observed that gradual unfreezing had no significant impact on the model's accuracy compared to standard fine-tuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

20 of 31 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2019How to Fine-Tune BERT for Text Classification?cited by 409
8 more
  1. 2017Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasmcited by 117
  2. 2019TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selectioncited by 74
  3. 2019Probing Biomedical Embeddings from Language Modelscited by 39
  4. 2017Making Neural QA as Simple as Possible but not Simplercited by 33
  5. 2019Pre-trained Language Model for Biomedical Question Answeringcited by 31
  6. 2017Neural Domain Adaptation for Biomedical Question Answeringcited by 30
  7. 2020Transferability of Natural Language Inference to Biomedical Question Answeringcited by 10
  8. 2019Measuring Domain Portability and ErrorPropagation in Biomedical QAcited by 3

Built on by

Not cited yet.