mlsanityData to 7 Oct 2026

Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review

arXiv 2510.17892 ↗First posted 18 Oct 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

The exponential increase in scientific literature and online information necessitates efficient methods for extracting knowledge from textual data. Natural language processing (NLP) plays a crucial role in addressing this challenge, particularly in text classification tasks. While large language models (LLMs) have achieved remarkable success in NLP, their accuracy can suffer in domain-specific contexts due to specialized vocabulary, unique grammatical structures, and imbalanced data distributions. In this systematic literature review (SLR), we investigate the utilization of pre-trained language models (PLMs) for domain-specific text classification. We systematically review 41 articles published between 2018 and January 2024, adhering to the PRISMA statement (preferred reporting items for systematic reviews and meta-analyses). This review methodology involved rigorous inclusion criteria and a multi-step selection process employing AI-powered tools. We delve into the evolution of text classification techniques and differentiate between traditional and modern approaches. We emphasize transformer-based models and explore the challenges and considerations associated with using LLMs for domain-specific text classification. Furthermore, we categorize existing research based on various PLMs and propose a taxonomy of techniques used in the field. To validate our findings, we conducted a comparative experiment involving BERT, SciBERT, and BioBERT in biomedical sentence classification. Finally, we present a comparative study on the performance of LLMs in text classification tasks across different domains. In addition, we examine recent advancements in PLMs for domain-specific text classification and offer insights into future directions and limitations in this rapidly evolving domain.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 76 references that are in the map. First posted on arXiv 18 Oct 2025.

Builds on

36 of 76 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  9. 2023BloombergGPT: A Large Language Model for Financecited by 923
  10. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
24 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  2. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  3. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  4. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  5. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 123
  6. 2023Text Classification via Large Language Modelscited by 104
  7. 2023Do We Still Need Clinical Language Models?cited by 97
  8. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  9. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  10. 2019SenseBERT: Driving Some Sense into BERTcited by 73
  11. 2020Comparing BERT against traditional machine learning text classificationcited by 45
  12. 2023Open, Closed, or Small Language Models for Text Classification?cited by 34
  13. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32
  14. 2023Exploring the Boundaries of GPT-4 in Radiologycited by 31
  15. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  16. 2023Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Taskscited by 25
  17. 2023Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithmscited by 15
  18. 2021Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERTcited by 14
  19. 2022SsciBERT: A Pre-trained Language Model for Social Science Textscited by 12
  20. 2020NukeBERT: A Pre-trained language model for Low Resource Nuclear Domaincited by 7
  21. 2021Self-interpretable Convolutional Neural Networks for Text Classificationcited by 5
  22. 2023Text2Topic: Multi-Label Text Classification System for Efficient Topic Detection in User Generated Content with Zero-Shot Capabilitiescited by 5
  23. 2021NukeLM: Pre-Trained and Fine-Tuned Language Models for the Nuclear and Energy Domainscited by 3
  24. 2023Leveraging Domain Knowledge for Inclusive and Bias-aware Humanitarian Response Entry Classificationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025From Chaos to Clarity: Schema-Constrained AI for Auditable Biomedical Evidence Extraction from Full-Text PDFscited by 1
  2. 2026AI Co-Scientist for Knowledge Synthesis in Medical Contexts: A Proof of Conceptcited by 0
  3. 2026Ensembles of Large Language Models for Identifying EQ-5D Studies in PubMed Based on Their Abstractscited by 0