Understanding the Challenges and Opportunities of Generative AI Apps: An Empirical Study

Generative AI (Gen-AI) is increasingly integrated into mobile applications (apps), introducing new capabilities while also creating new challenges for users. However, despite their growing adoption, we lack an ecosystem-level understanding of the experiences, opportunities, and challenges users report across Gen-AI mobile apps. We conduct a user-centered analysis of 1,035,342 reviews from 171 Gen-AI apps from the Google Play Store. We propose SARA (Selection, Acquisition, Refinement, and Analysis), a four-phase framework that leverages prompt-based LLMs for large-scale review analysis. We validate the reliability of LLM-based topic extraction and assignment using 4,353 manually evaluated reviews, achieving 91% accuracy with five-shot prompting and filtering of non-informative reviews. We identify the top ten topics (e.g., AI Performance and Emotional Connection) and perform a cross-platform comparison with Apple App Store reviews. Through qualitative analysis of 762 reviews, we uncover three opportunities (AI for Accessibility and Wellbeing, AI as a Collaborative Creative Tool, and AI Versatility) and three challenges (Managing User Expectations and AI Limitations, Balancing Content Moderation and Creative Freedom, and Strategic Integration of Gen-AI Features). Finally, we analyze temporal trends, revealing how the topics discussed and their evaluations change over time, including changing concerns around emotional connection and content moderation.