# Automating Categorization of Scientific Texts with In-Context Learning and Prompt-Chaining in Large Language Models

arXiv: https://arxiv.org/abs/2604.23430 · HTML: https://mlsanity.com/paper/2604.23430/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 25 Apr 2026.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [The Semantic Scholar Open Data Platform](https://mlsanity.com/paper/2301.10140/) (2023)
- [Regulating ChatGPT and other Large Generative AI Models](https://mlsanity.com/paper/2302.02337/) (2023)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases](https://mlsanity.com/paper/2012.07675/) (2020)
- [FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19](https://mlsanity.com/paper/2006.11343/) (2020)
- [Language Models as Black-Box Optimizers for Vision-Language Models](https://mlsanity.com/paper/2309.05950/) (2023)
- [Generative AI Text Classification using Ensemble LLM Approaches](https://mlsanity.com/paper/2309.07755/) (2023)
- [LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis](https://mlsanity.com/paper/2409.18812/) (2024)
- [A Review of the Challenges with Massive Web-mined Corpora Used in Large Language Models Pre-Training](https://mlsanity.com/paper/2407.07630/) (2024)
- [Improving Scholarly Knowledge Representation: Evaluating BERT-based Models for Scientific Relation Classification](https://mlsanity.com/paper/2004.06153/) (2020)
- [Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance](https://mlsanity.com/paper/2412.17321/) (2024)
- [On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts](https://mlsanity.com/paper/2502.15745/) (2025)
- [Enhancing Research Information Systems with Identification of Domain Experts](https://mlsanity.com/paper/2404.02921/) (2024)

## Abstract (arXiv)

The relentless expansion of scientific literature presents significant challenges for navigation and knowledge discovery. Within Research Information Retrieval, established tasks such as text summarization and classification remain crucial for enabling researchers and practitioners to effectively navigate this vast landscape, so that efforts have increasingly been focused on developing advanced research information systems. These systems aim not only to provide standard keyword-based search functionalities but also to incorporate capabilities for automatic content categorization within knowledge-intensive organizations across academia and industry. This study systematically evaluates the performance of off-the-shelf Large Language Models (LLMs) in analyzing scientific texts according to a given classification scheme. We utilized the hierarchical ORKG taxonomy as a classification framework, employing the FORC dataset as ground truth. We investigated the effectiveness of advanced prompt engineering strategies, namely In-Context Learning (ICL) and Prompt Chaining, and experimentally explored the influence of the LLMs' temperature hyperparameter on classification accuracy. Our experiments demonstrate that Prompt Chaining yields superior classification accuracy compared to pure ICL, particularly when applied to the nested structure of the ORKG taxonomy. LLMs with prompt chaining outperform the state-of-the-art models for domain (1st level) prediction and show even better performance for subject (2nd level) prediction compared to the older BERT model. However, LLMs are not yet able to perform well in classifying the topic (3rd level) of research areas based on this specific hierarchical taxonomy, as they only reach about 50% accuracy even with prompt chaining.