# Exploring Qualitative Research Using LLMs

arXiv: https://arxiv.org/abs/2306.13298 · HTML: https://mlsanity.com/paper/2306.13298/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 48 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 7 of 7)

- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [A review of possible effects of cognitive biases on the interpretation of rule-based machine learning models](https://mlsanity.com/paper/1804.02969/) (2018)
- [Diversity and Inclusion in Artificial Intelligence](https://mlsanity.com/paper/2305.12728/) (2023)
- [AI Ethics Principles in Practice: Perspectives of Designers and Developers](https://mlsanity.com/paper/2112.07467/) (2021)
- [Does Siri Have a Soul? Exploring Voice Assistants Through Shinto Design Fictions](https://mlsanity.com/paper/2003.03207/) (2020)
- [Human Values in Mobile App Development: An Empirical Study on Bangladeshi Agriculture Mobile Apps](https://mlsanity.com/paper/2110.05150/) (2021)

## Built on by (most cited first, 5 of 5)

- [Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies](https://mlsanity.com/paper/2309.13063/) (2023)
- [Investigating Responsible AI for Scientific Research: An Empirical Study](https://mlsanity.com/paper/2312.09561/) (2023)
- [Large Language Model for Qualitative Research -- A Systematic Mapping Study](https://mlsanity.com/paper/2411.14473/) (2024)
- [Morality is Contextual: Learning Interpretable Moral Contexts from Human Data with Probabilistic Clustering and Large Language Models](https://mlsanity.com/paper/2512.21439/) (2025)
- [Large Language Models and Thematic Analysis: Human-AI Synergy in Researching Hate Speech on Social Media](https://mlsanity.com/paper/2408.05126/) (2024)

## Abstract (arXiv)

The advent of AI driven large language models (LLMs) have stirred discussions about their role in qualitative research. Some view these as tools to enrich human understanding, while others perceive them as threats to the core values of the discipline. This study aimed to compare and contrast the comprehension capabilities of humans and LLMs. We conducted an experiment with small sample of Alexa app reviews, initially classified by a human analyst. LLMs were then asked to classify these reviews and provide the reasoning behind each classification. We compared the results with human classification and reasoning. The research indicated a significant alignment between human and ChatGPT 3.5 classifications in one third of cases, and a slightly lower alignment with GPT4 in over a quarter of cases. The two AI models showed a higher alignment, observed in more than half of the instances. However, a consensus across all three methods was seen only in about one fifth of the classifications. In the comparison of human and LLMs reasoning, it appears that human analysts lean heavily on their individual experiences. As expected, LLMs, on the other hand, base their reasoning on the specific word choices found in app reviews and the functional components of the app itself. Our results highlight the potential for effective human LLM collaboration, suggesting a synergistic rather than competitive relationship. Researchers must continuously evaluate LLMs role in their work, thereby fostering a future where AI and humans jointly enrich qualitative research.