# Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?

arXiv: https://arxiv.org/abs/2407.12626 · HTML: https://mlsanity.com/paper/2407.12626/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016)
- [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018)
- [Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis](https://mlsanity.com/paper/2210.04714/) (2022)
- [DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks](https://mlsanity.com/paper/1906.04569/) (2019)
- [Quantifying Uncertainties in Natural Language Processing Tasks](https://mlsanity.com/paper/1811.07253/) (2018)
- [Learning to select data for transfer learning with Bayesian Optimization](https://mlsanity.com/paper/1707.05246/) (2017)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law](https://mlsanity.com/paper/2209.06049/) (2022)
- [Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches](https://mlsanity.com/paper/2211.16285/) (2022)
- [Neural Domain Adaptation for Biomedical Question Answering](https://mlsanity.com/paper/1706.03610/) (2017)
- [FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain](https://mlsanity.com/paper/2304.04280/) (2023)

## Abstract (arXiv)

The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in-chief of which is a model's ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model's output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.