# Exploring Zero-Shot App Review Classification with ChatGPT: Challenges and Potential

arXiv: https://arxiv.org/abs/2505.04759 · HTML: https://mlsanity.com/paper/2505.04759/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 25 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 4 of 4)

- [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://mlsanity.com/paper/2307.11760/) (2023)
- [Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers](https://mlsanity.com/paper/2306.04504/) (2023)
- [Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews](https://mlsanity.com/paper/2104.05861/) (2021)
- [A Critical Review of Large Language Models: Sensitivity, Bias, and the Path Toward Specialized AI](https://mlsanity.com/paper/2307.15425/) (2023)

## Built on by (most cited first, 1 of 1)

- [Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy](https://mlsanity.com/paper/2603.05637/) (2026)

## Abstract (arXiv)

App reviews are a critical source of user feedback, offering valuable insights into an app's performance, features, usability, and overall user experience. Effectively analyzing these reviews is essential for guiding app development, prioritizing feature updates, and enhancing user satisfaction. Classifying reviews into functional and non-functional requirements play a pivotal role in distinguishing feedback related to specific app features (functional requirements) from feedback concerning broader quality attributes, such as performance, usability, and reliability (non-functional requirements). Both categories are integral to informed development decisions. Traditional approaches to classifying app reviews are hindered by the need for large, domain-specific datasets, which are often costly and time-consuming to curate. This study explores the potential of zero-shot learning with ChatGPT for classifying app reviews into four categories: functional requirement, non-functional requirement, both, or neither. We evaluate ChatGPT's performance on a benchmark dataset of 1,880 manually annotated reviews from ten diverse apps spanning multiple domains. Our findings demonstrate that ChatGPT achieves a robust F1 score of 0.842 in review classification, despite certain challenges and limitations. Additionally, we examine how factors such as review readability and length impact classification accuracy and conduct a manual analysis to identify review categories more prone to misclassification.