# Advancing Scientific Text Classification: Fine-Tuned Models with Dataset Expansion and Hard-Voting

arXiv: https://arxiv.org/abs/2504.19021 · HTML: https://mlsanity.com/paper/2504.19021/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 26 Apr 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [HDLTex: Hierarchical Deep Learning for Text Classification](https://mlsanity.com/paper/1709.08267/) (2017)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction](https://mlsanity.com/paper/2109.15290/) (2021)
- [Structured information extraction from complex scientific text with fine-tuned large language models](https://mlsanity.com/paper/2212.05238/) (2022)
- [A Survey of Active Learning for Text Classification using Deep Neural Networks](https://mlsanity.com/paper/2008.07267/) (2020)
- [Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks](https://mlsanity.com/paper/1603.03827/) (2016)
- [Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT](https://mlsanity.com/paper/2107.04374/) (2021)
- [Fine-Tuning Large Language Models for Scientific Text Classification: A Comparative Study](https://mlsanity.com/paper/2412.00098/) (2024)
- [Towards Data-efficient Customer Intent Recognition with Prompt-based Learning Paradigm](https://mlsanity.com/paper/2309.14779/) (2023)

## Abstract (arXiv)

Efficient text classification is essential for handling the increasing volume of academic publications. This study explores the use of pre-trained language models (PLMs), including BERT, SciBERT, BioBERT, and BlueBERT, fine-tuned on the Web of Science (WoS-46985) dataset for scientific text classification. To enhance performance, we augment the dataset by executing seven targeted queries in the WoS database, retrieving 1,000 articles per category aligned with WoS-46985's main classes. PLMs predict labels for this unlabeled data, and a hard-voting strategy combines predictions for improved accuracy and confidence. Fine-tuning on the expanded dataset with dynamic learning rates and early stopping significantly boosts classification accuracy, especially in specialized domains. Domain-specific models like SciBERT and BioBERT consistently outperform general-purpose models such as BERT. These findings underscore the efficacy of dataset augmentation, inference-driven label prediction, hard-voting, and fine-tuning techniques in creating robust and scalable solutions for automated academic text classification.