mlsanityData to 14 Oct 2026

LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis

arXiv 2310.15100 ↗First posted 23 Oct 2023cs.CLcited by 70

Not on the map. Why?

Abstract (arXiv)

Thematic analysis (TA) has been widely used for analyzing qualitative data in many disciplines and fields. To ensure reliable analysis, the same piece of data is typically assigned to at least two human coders. Moreover, to produce meaningful and useful analysis, human coders develop and deepen their data interpretation and coding over multiple iterations, making TA labor-intensive and time-consuming. Recently the emerging field of large language models (LLMs) research has shown that LLMs have the potential replicate human-like behavior in various tasks: in particular, LLMs outperform crowd workers on text-annotation tasks, suggesting an opportunity to leverage LLMs on TA. We propose a human-LLM collaboration framework (i.e., LLM-in-the-loop) to conduct TA with in-context learning (ICL). This framework provides the prompt to frame discussions with a LLM (e.g., GPT-3.5) to generate the final codebook for TA. We demonstrate the utility of this framework using survey datasets on the aspects of the music listening experience and the usage of a password manager. Results of the two case studies show that the proposed framework yields similar coding quality to that of human coders but reduces TA's labor and time demands.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Builds on 14 of its 39 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

14 of 39 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  12. 2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
2 more
  1. 2020Deep Sentiment Classification and Topic Discovery on Novel Coronavirus or COVID-19 Online Discussions: NLP Using LSTM Recurrent Neural Network Approachcited by 20
  2. 2023Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the modelcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 25; 2025: 21; 2026: 22.

70 papers.

Sort:

By area:

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  2. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 42
  3. 2024VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Contextcited by 31
  4. 2024Large Language Models for Manufacturingcited by 24
  5. 2024Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Taskscited by 22
  6. 2024Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysiscited by 20
  7. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  8. 2025HICode: Hierarchical Inductive Coding with LLMscited by 12
  9. 2024Automating Thematic Analysis: How LLMs Analyse Controversial Topicscited by 10
  10. 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
  11. 2023From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviewscited by 9
  12. 2024Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategycited by 9
38 more
  1. 2024Interpreting Language Reward Models via Contrastive Explanationscited by 9
  2. 2025TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviewscited by 9
  3. 2025Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learningcited by 9
  4. 2024Discovering Latent Themes in Social Media Messaging: A Machine-in-the-Loop Approach Integrating LLMscited by 8
  5. 2024QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forumscited by 8
  6. 2024Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AIcited by 6
  7. 2024A Computational Method for Measuring "Open Codes" in Qualitative Analysiscited by 6
  8. 2025Humanlike AI Design Increases Anthropomorphism but Yields Divergent Outcomes on Engagement and Trust Globallycited by 6
  9. 2024CoDefeater: Using LLMs To Find Defeaters in Assurance Casescited by 5
  10. 2025Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answeringcited by 5
  11. 2024Investigating Expert-in-the-Loop LLM Discourse Patterns for Ancient Intertextual Analysiscited by 4
  12. 2024Post-hoc Study of Climate Microtargeting on Social Media Ads with LLMs: Thematic Insights and Fairness Evaluationcited by 4
  13. 2024Analyzing Nobel Prize Literature with Large Language Modelscited by 4
  14. 2025Value Alignment of Social Media Ranking Algorithmscited by 4
  15. 2023Grounded Intuition of GPT-Vision's Abilities with Scientific Imagescited by 3
  16. 2024LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languagescited by 3
  17. 2026Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Riskscited by 3
  18. 2024Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysiscited by 2
  19. 2025LLM-Assisted Automated Deductive Coding of Dialogue Data: Leveraging Dialogue-Specific Characteristics to Enhance Contextual Understandingcited by 2
  20. 2025MuseScorer: Idea Originality Scoring At Scalecited by 2
  21. 2025What is a protest anyway? Codebook conceptualization is still a first-order concern in LLM-era classificationcited by 2
  22. 2026Like a Therapist, But Not: Reddit Narratives of AI in Mental Health Contextscited by 2
  23. 2024Can Large Language Models Create New Knowledge for Spatial Reasoning Tasks?cited by 1
  24. 2024Training LLMs to Recognize Hedges in Spontaneous Narrativescited by 1
  25. 2025LLM-TA: An LLM-Enhanced Thematic Analysis Pipeline for Transcripts from Parents of Children with Congenital Heart Diseasecited by 1
  26. 2025Comparing Native and Non-native English Speakers' Behaviors in Collaborative Writing through Visual Analyticscited by 1
  27. 2025Invisible Walls in Cities: Designing LLM Agent to Predict Urban Segregation Experience with Social Media Contentcited by 1
  28. 2025Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: An Empirical Study on Popular Open-Source Projectscited by 1
  29. 2025Human vs. LLM-Based Thematic Analysis for Digital Mental Health Research: Proof-of-Concept Comparative Studycited by 1
  30. 2025Foundation of Intelligence: Review of Math Word Problems from Human Cognition Perspectivecited by 1
  31. 2026What happens when reviewers receive AI feedback in their reviews?cited by 1
  32. 2026How Trustworthy Are LLM-as-Judge Ratings for Interpretive Responses? Implications for Qualitative Research Workflowscited by 1
  33. 2026Mitigating Prompt-Induced Cognitive Biases in General-Purpose AI for Software Engineeringcited by 1
  34. 2026Agent-as-Peer-Debriefer: A Multi-Agent Framework with Perspective-Based Refinement for Qualitative Analysiscited by 1
  35. 2024SLEGO: A Collaborative Data Analytics System with LLM Recommender for Diverse Userscited by 0
  36. 2024Iterative Improvement of an Additively Regularized Topic Modelcited by 0
  37. 2024Effects of Collaboration on the Performance of Interactive Theme Discovery Systemscited by 0
  38. 2024MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Taskscited by 0

50 most cited of 70 All 70 (JSON)