Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) offer new possibilities for scaling qualitative analysis, but existing applications often provide limited methodological transparency regarding how qualitative methods are translated into computational procedures. This paper presents an auditable and privacy-preserving computational operationalization of inductive and latent Thematic Analysis (TA). This paper first derives five design principles from the methodological requirements of TA and the conditions introduced by LLM-based inference: preserving interpretative context, maintaining traceable relationships between empirical material and analytical outputs, representing analytical constructs and reasoning explicitly, constraining LLM inference to interpretative tasks, and enabling privacy-preserving local deployment. Second, it presents a proof-of-concept for a two-phase workflow that operationalizes these principles by combining interpretative LLM inference with deterministic procedural control to generate codes, analytical justifications, themes, and theme descriptions while preserving explicit links to the source material. Third, it proposes an evaluation framework combining structural comparison with human-led TA and independent expert assessment of analytical quality. The evaluation is conducted on semi-structured Danish interview transcripts. and the results shows that the workflow produces code-level outputs with coverage broadly comparable to human annotations and highly rated analytical justifications, while generating a more compressed thematic structure characterized by fewer and broader themes. The findings demonstrate the feasibility of auditable LLM-supported TA through a modular workflow designed to scale to larger datasets, accommodate different LLMs, and support transfer across research domains, with domain adaptation primarily requiring adjustments to the prompting strategy.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 6 of its 23 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
6 of 23 references.
References · 6
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
- 2021Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systemscited by 26
- 2026Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluationcited by 13
- 2025Large Language Models in Thematic Analysis: Prompt Engineering, Evaluation, and Guidelines for Qualitative Software Engineering Researchcited by 5
- 2025LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendationscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Automating Thematic Analysis: How LLMs Analyse Controversial Topics
- 2024Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis
- 2024Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks
- 2025LLMCode: Evaluating and Enhancing Researcher-AI Alignment in Qualitative Analysis ↗
- 2026Large Language Models as Supervised Extraction Assistants: Lowering the Barrier to Documentation Standard Adoption in Agent-Based Modelling
- 2024LLM Augmentations to support Analytical Reasoning over Multiple Documents
- 2023Exploring Qualitative Research Using LLMs
- 2026Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods