# Students Need More Attention: BERT-based AttentionModel for Small Data with Application to AutomaticPatient Message Triage

arXiv: https://arxiv.org/abs/2006.11991 · HTML: https://mlsanity.com/paper/2006.11991/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2020.

## Built on by (most cited first, 16 of 16)

- [CLIP: A Dataset for Extracting Action Items for Physicians from Hospital Discharge Notes](https://mlsanity.com/paper/2106.02524/) (2021)
- [Federated Split BERT for Heterogeneous Text Classification](https://mlsanity.com/paper/2205.13299/) (2022)
- ["Define Your Terms" : Enhancing Efficient Offensive Speech Classification with Definition](https://mlsanity.com/paper/2402.03221/) (2024)
- [Federated Non-negative Matrix Factorization for Short Texts Topic Modeling with Mutual Information](https://mlsanity.com/paper/2205.13300/) (2022)
- [Medical Triage as Pairwise Ranking: A Benchmark for Urgency in Patient Portal Messages](https://mlsanity.com/paper/2601.13178/) (2026)
- [In-Context Learning for Preserving Patient Privacy: A Framework for Synthesizing Realistic Patient Portal Messages](https://mlsanity.com/paper/2411.06549/) (2024)
- [KDH-MLTC: Knowledge Distillation for Healthcare Multi-Label Text Classification](https://mlsanity.com/paper/2505.07162/) (2025)
- [Global Readiness of Language Technology for Healthcare: What would it Take to Combat the Next Pandemic?](https://mlsanity.com/paper/2204.02790/) (2022)
- [A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Tasks](https://mlsanity.com/paper/2212.05251/) (2022)
- [Few-Shot Large Language Models for Actionable Triage Categorization of Online Patient Inquiries](https://mlsanity.com/paper/2605.15680/) (2026)
- [VU-BERT: A Unified framework for Visual Dialog](https://mlsanity.com/paper/2202.10787/) (2022)
- [DEFTri: A Few-Shot Label Fused Contextual Representation Learning For Product Defect Triage in e-Commerce](https://mlsanity.com/paper/2307.11344/) (2023)
- [Revisiting the Role of Label Smoothing in Enhanced Text Sentiment Classification](https://mlsanity.com/paper/2312.06522/) (2023)
- [Exploring the Capabilities of ChatGPT in Ancient Chinese Translation and Person Name Recognition](https://mlsanity.com/paper/2312.15304/) (2023)
- [Extracting Post-Acute Sequelae of SARS-CoV-2 Infection Symptoms from Clinical Notes via Hybrid Natural Language Processing](https://mlsanity.com/paper/2508.12405/) (2025)
- [AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment](https://mlsanity.com/paper/2605.11398/) (2026)

## Abstract (arXiv)

Small and imbalanced datasets commonly seen in healthcare represent a challenge when training classifiers based on deep learning models. So motivated, we propose a novel framework based on BioBERT (Bidirectional Encoder Representations from Transformers forBiomedical TextMining). Specifically, (i) we introduce Label Embeddings for Self-Attention in each layer of BERT, which we call LESA-BERT, and (ii) by distilling LESA-BERT to smaller variants, we aim to reduce overfitting and model size when working on small datasets. As an application, our framework is utilized to build a model for patient portal message triage that classifies the urgency of a message into three categories: non-urgent, medium and urgent. Experiments demonstrate that our approach can outperform several strong baseline classifiers by a significant margin of 4.3% in terms of macro F1 score. The code for this project is publicly available at urlhttps://github.com/shijing001/textclassifiers.