# On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts

arXiv: https://arxiv.org/abs/2502.15745 · HTML: https://mlsanity.com/paper/2502.15745/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 8 Feb 2025.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [The Semantic Scholar Open Data Platform](https://mlsanity.com/paper/2301.10140/) (2023)
- [Regulating ChatGPT and other Large Generative AI Models](https://mlsanity.com/paper/2302.02337/) (2023)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases](https://mlsanity.com/paper/2012.07675/) (2020)
- [FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19](https://mlsanity.com/paper/2006.11343/) (2020)
- [Language Models as Black-Box Optimizers for Vision-Language Models](https://mlsanity.com/paper/2309.05950/) (2023)
- [Generative AI Text Classification using Ensemble LLM Approaches](https://mlsanity.com/paper/2309.07755/) (2023)
- [LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis](https://mlsanity.com/paper/2409.18812/) (2024)
- [A Review of the Challenges with Massive Web-mined Corpora Used in Large Language Models Pre-Training](https://mlsanity.com/paper/2407.07630/) (2024)
- [Improving Scholarly Knowledge Representation: Evaluating BERT-based Models for Scientific Relation Classification](https://mlsanity.com/paper/2004.06153/) (2020)
- [Enhancing Research Information Systems with Identification of Domain Experts](https://mlsanity.com/paper/2404.02921/) (2024)
- [Fine-tuning and Prompt Engineering with Cognitive Knowledge Graphs for Scholarly Knowledge Organization](https://mlsanity.com/paper/2409.06433/) (2024)

## Built on by (most cited first, 3 of 3)

- [Defining, Understanding, and Detecting Online Toxicity: Challenges and Machine Learning Approaches](https://mlsanity.com/paper/2509.14264/) (2025)
- [Towards Structured Knowledge: Advancing Triple Extraction from Regional Trade Agreements using Large Language Models](https://mlsanity.com/paper/2510.05121/) (2025)
- [Automating Categorization of Scientific Texts with In-Context Learning and Prompt-Chaining in Large Language Models](https://mlsanity.com/paper/2604.23430/) (2026)

## Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) has led to a multitude of application opportunities. One traditional task for Information Retrieval systems is the summarization and classification of texts, both of which are important for supporting humans in navigating large literature bodies as they e.g. exist with scientific publications. Due to this rapidly growing body of scientific knowledge, recent research has been aiming at building research information systems that not only offer traditional keyword search capabilities, but also novel features such as the automatic detection of research areas that are present at knowledge intensive organizations in academia and industry. To facilitate this idea, we present the results obtained from evaluating a variety of LLMs in their ability to sort scientific publications into hierarchical classifications systems. Using the FORC dataset as ground truth data, we have found that recent LLMs (such as Meta Llama 3.1) are able to reach an accuracy of up to 0.82, which is up to 0.08 better than traditional BERT models.