# Actively evaluating and learning the distinctions that matter: Vaccine safety signal detection from emergency triage notes

arXiv: https://arxiv.org/abs/2507.18123 · HTML: https://mlsanity.com/paper/2507.18123/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 43 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Builds on (most cited first, 9 of 9)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [FlipDA: Effective and Robust Data Augmentation for Few-Shot Learning](https://mlsanity.com/paper/2108.06332/) (2021)
- [Medical symptom recognition from patient text: An active learning approach for long-tailed multilabel distributions](https://mlsanity.com/paper/2011.06874/) (2020)
- [ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms](https://mlsanity.com/paper/2104.04999/) (2021)

## Abstract (arXiv)

The rapid development of COVID-19 vaccines has showcased the global communitys ability to combat infectious diseases. However, the need for post-licensure surveillance systems has grown due to the limited window for safety data collection in clinical trials and early widespread implementation. This study aims to employ Natural Language Processing techniques and Active Learning to rapidly develop a classifier that detects potential vaccine safety issues from emergency department notes. ED triage notes, containing expert, succinct vital patient information at the point of entry to health systems, can significantly contribute to timely vaccine safety signal surveillance. While keyword-based classification can be effective, it may yield false positives and demand extensive keyword modifications. This is exacerbated by the infrequency of vaccination-related ED presentations and their similarity to other reasons for ED visits. NLP offers a more accurate and efficient alternative, albeit requiring annotated data, which is often scarce in the medical field. Active learning optimizes the annotation process and the quality of annotated data, which can result in faster model implementation and improved model performance. This work combines active learning, data augmentation, and active learning and evaluation techniques to create a classifier that is used to enhance vaccine safety surveillance from ED triage notes.