mlsanityData to 14 Oct 2026

Lightweight Baselines for Medical Abstract Classification: DistilBERT with Cross-Entropy as a Strong Default

arXiv 2510.10025 ↗First posted 11 Oct 2025cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

The research evaluates lightweight medical abstract classification methods to establish their maximum performance capabilities under financial budget restrictions. On the public medical abstracts corpus, we finetune BERT base and Distil BERT with three objectives cross entropy (CE), class weighted CE, and focal loss under identical tokenization, sequence length, optimizer, and schedule. DistilBERT with plain CE gives the strongest raw argmax trade off, while a post hoc operating point selection (validation calibrated, classwise thresholds) sub stantially improves deployed performance; under this tuned regime, focal benefits most. We report Accuracy, Macro F1, and WeightedF1, release evaluation artifacts, and include confusion analyses to clarify error structure. The practical takeaway is to start with a compact encoder and CE, then add lightweight calibration or thresholding when deployment requires higher macro balance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

22 of 42 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  6. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  7. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  8. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  9. 2020Long-tail learning via logit adjustmentcited by 618
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  11. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 513
  12. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
10 more
  1. 2020Calibration of Pre-trained Transformerscited by 315
  2. 2024Swift Sampler: Efficient Learning of Sampler by 10 Parameterscited by 25
  3. 2025Does Bigger Mean Better? Comparitive Analysis of CNNs and Biomedical Vision Language Modles in Medical Diagnosiscited by 9
  4. 2025MemeBLIP2: A novel lightweight multimodal system to detect harmful memescited by 8
  5. 2025Applications of Large Models in Medicinecited by 6
  6. 2025Rainbow Noise: Stress-Testing Multimodal Harmful-Meme Detectors on LGBTQ Contentcited by 6
  7. 2025No Black Boxes: Interpretable and Interactable Predictive Healthcare with Knowledge-Enhanced Agentic Causal Discoverycited by 5
  8. 2025Modeling Parkinson's Disease Progression Using Longitudinal Voice Biomarkers: A Comparative Study of Statistical and Neural Mixed-Effects Modelscited by 5
  9. 2024Bridging Stepwise Lab-Informed Pretraining and Knowledge-Guided Learning for Diagnostic Reasoningcited by 3
  10. 2025Toward Causal-Visual Programming: Enhancing Agentic Reasoning in Low-Code Environmentscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuningcited by 1
  2. 2025Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignmentcited by 0
  3. 2026Physiologically Informed Deep Learning: A Multi-Scale Framework for Next-Generation PBPK Modelingcited by 0