# Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding

arXiv: https://arxiv.org/abs/2304.10548 · HTML: https://mlsanity.com/paper/2304.10548/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 7 of 7)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation](https://mlsanity.com/paper/2209.11000/) (2022)
- [Conversational agents for fostering curiosity-driven learning in children](https://mlsanity.com/paper/2204.03546/) (2022)

## Built on by (most cited first, 30 of 82)

- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions](https://mlsanity.com/paper/2412.05563/) (2024)
- [Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM](https://mlsanity.com/paper/2404.12259/) (2024)
- [LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis](https://mlsanity.com/paper/2310.15100/) (2023)
- [LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding](https://mlsanity.com/paper/2306.14924/) (2023)
- ['Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants](https://mlsanity.com/paper/2409.19430/) (2024)
- [Embedding Democratic Values into Social Media AIs via Societal Objective Functions](https://mlsanity.com/paper/2307.13912/) (2023)
- [LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Model](https://mlsanity.com/paper/2310.04445/) (2023)
- [Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis](https://mlsanity.com/paper/2402.01386/) (2024)
- [Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model](https://mlsanity.com/paper/2305.13014/) (2023)
- [A Systematic Evaluation of LLM Strategies for Mental Health Text Analysis: Fine-tuning vs. Prompt Engineering vs. RAG](https://mlsanity.com/paper/2503.24307/) (2025)
- [HICode: Hierarchical Inductive Coding with LLMs](https://mlsanity.com/paper/2509.17946/) (2025)
- [Automating Thematic Analysis: How LLMs Analyse Controversial Topics](https://mlsanity.com/paper/2405.06919/) (2024)
- [Exploring the Use of Abusive Generative AI Models on Civitai](https://mlsanity.com/paper/2407.12876/) (2024)
- [From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews](https://mlsanity.com/paper/2312.01202/) (2023)
- [Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits](https://mlsanity.com/paper/2403.14791/) (2024)
- [Interpreting Language Reward Models via Contrastive Explanations](https://mlsanity.com/paper/2411.16502/) (2024)
- [From Conversation to Orchestration: HCI Challenges and Opportunities in Interactive Multi-Agentic Systems](https://mlsanity.com/paper/2506.20091/) (2025)
- [Harnessing ChatGPT for thematic analysis: Are we ready?](https://mlsanity.com/paper/2310.14545/) (2023)
- [Predicting challenge moments from students' discourse: A comparison of GPT-4 to two traditional natural language processing approaches](https://mlsanity.com/paper/2401.01692/) (2024)
- [QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums](https://mlsanity.com/paper/2405.05345/) (2024)
- [Writing user personas with Large Language Models: Testing phase 6 of a Thematic Analysis of semi-structured interviews](https://mlsanity.com/paper/2305.18099/) (2023)
- [Can GPT-4 Replicate Empirical Software Engineering Research?](https://mlsanity.com/paper/2310.01727/) (2023)
- [Generative Artificial Intelligence in Learning Analytics: Contextualising Opportunities and Challenges through the Learning Analytics Cycle](https://mlsanity.com/paper/2312.00087/) (2023)
- [Reflections on Inductive Thematic Saturation as a potential metric for measuring the validity of an inductive Thematic Analysis with LLMs](https://mlsanity.com/paper/2401.03239/) (2024)
- [A Computational Method for Measuring "Open Codes" in Qualitative Analysis](https://mlsanity.com/paper/2411.12142/) (2024)
- [The Evolution of LLM Adoption in Industry Data Curation Practices](https://mlsanity.com/paper/2412.16089/) (2024)
- [Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors](https://mlsanity.com/paper/2409.18203/) (2024)
- [Labeling Free-text Data using Language Model Ensembles](https://mlsanity.com/paper/2501.08413/) (2025)
- [Using Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teaching](https://mlsanity.com/paper/2403.11984/) (2024)

## Abstract (arXiv)

Qualitative analysis of textual contents unpacks rich and valuable information by assigning labels to the data. However, this process is often labor-intensive, particularly when working with large datasets. While recent AI-based tools demonstrate utility, researchers may not have readily available AI resources and expertise, let alone be challenged by the limited generalizability of those task-specific models. In this study, we explored the use of large language models (LLMs) in supporting deductive coding, a major category of qualitative analysis where researchers use pre-determined codebooks to label the data into a fixed set of codes. Instead of training task-specific models, a pre-trained LLM could be used directly for various tasks without fine-tuning through prompt learning. Using a curiosity-driven questions coding task as a case study, we found, by combining GPT-3 with expert-drafted codebooks, our proposed approach achieved fair to substantial agreements with expert-coded results. We lay out challenges and opportunities in using LLMs to support qualitative coding and beyond.